Story Arbiter of ash- what happens if I leave?

First time I made it to the arbiter of ash. I understand the mechanics now but I’m to tired for his timings. What happens if I leave the boss fight now?
Do I need to collect the 2 fragments again or can I just go there at another time?
It’s when you fight him for the first time in the quest line.