Do the villagers visit the deep woods?

Caldarus always mentions seeing some villagers there, but I never met anyone out in that area. I imagined they would go out to the deep woods eventually, especially the people who have family buried in the cemetery - there was even dialogue about it - but I've never seen anyone in that place.