Do you always play with farmers or do you avoid using them?

I've noticed that whenever I play with my friends they avoid or boycott using farmers as they don't really like how OP they can be and that they prefer just gaining points with roads, cities and monasteries. Have you ever encountered this as well? I don't mind not using farmers myself so I'm ok with it, but it's interesting that most of the people I've played with just said right from the beginning that they would rather play without this rule.