Harm of washing clay down a sink?

What is the main concern about washing clay down your sink? Clay in the septic system or clay in the pipes? I have a utility sink in my house that drains into my field about 50 yards from my house. I am wondering if washing small amounts of clay down it, would really cause an issue.