By using fossilized domesticated crops as evidence for agriculture, aren't we ignoring cultures who practiced agriculture before modern crops developped?

Ex. I asked a question here about how we knew that the people who made gobekli tepe didn't have agriculture. The answer I got was essentially that we haven't found any remains or evidence of modern domesticated crops near the site.

My question is, presumably the first people to use agriculture had nothing to work with other than wild crops, so why is the lack of domesticated crop fossils considered as proof that an ancient people didn't practice agriculture?