"Pottery Material"

I'm working on the star path and one of the tasks is "Dig for Pottery Material". I'm assuming that means clay. Which can be found in the Forgotten Lands, Sunlit Plateau, and Glade of Trust. It's a very slow process. I've dug a ton of random holes over 2 or 3 days and only have 7 of the 20 I need. I'm even using a person whose specialty is digging, but it seems that the "extras" that the character givea me doesn't count. Am I missing something? Is there a way to expedite this task? Or do I just suck it up and dig 20 holes in each area a day until I get to what I need?