Is it okay to use sub-quality soil in the bottom of a raised garden bed?

I edged our lawn and ended up with a huge trash bin full of dirt that's really dry and dusty. A lot of it is stuff the golfers have brought up over time. I guess it's full of grass seed because this whole area of the lawn was dead for the longest time but we've been getting rain and now the whole yard is growing grass. There's also probably tons of weed seed in it. I am considering using it in some small wooden containers that I got from work as fill because they are super deep (well over a foot) and I don't have a lot of gardening soil left (we buy from a local vendor, it's not bagged stuff).

So, what would be the fallout of using this at the bottom of the containers and then putting 6-12" of quality soil on top? Thank you!