How servants flip control points?

I asked this in steam forums too but did not get satisfying answer. Thing is im trying to unify last balkan countries to EU, but servants can just use magic and flip the crontrol point. No matter of public support or if the point is secured, they just turn it.

Im also super positive there are no one in the country! I have used survey area and there are non but my agents in the area when flip happens. if servant point comes defended next turn, i will detect councillor doing it. How are they doing it ad how to defend against it?