How to know when soil gnats pose a problem?

Sometimes here and there I'll see a gnat or two around my collection but usually never more than that at a sustained level. I know their babies can harm my plant roots, but should I start taking action at this level or not since there is never a large amount of them on my collection. [Adult form] And the damage that the larvae causes, how do you notice it? All my plants seem to be healthy currently.