What to do about soot post-sweep?

had a warm day yesterday and took it as an opportunity to clean the chimney. found some light creosote build up on the cap but the liner (6" stainless steel) had only a light coating of powdery soot. Gave it a good sweep with a poly

pro brush and of course am left with a pile of soot in the box.

I've been shoveling the soot out of the box as best as I can but it's now mixed with the ash and can't be entirely removed unless I just empty the box of all ash.

Do I need to remove all the soot? I can always throw some ash back into the box from my ash bucket, but it'll be a pain in the ass and probably messy. Or, can I just light a fire and not be concerned with some soot mixed with the ash in the box?

TIA