Where are all the men?

Sorry if this has been posted on this sub before but I'm just doing a quick rewatch of the whole show before the new season comes out. I also haven't read the books yet. I'm on season 2 where Emily has been sent to the colonies and I'm wondering... where did all the men go?

I know that women either became mistresses, handmaids, marthas, aunts, sex workers, or were killed, or sent to the colonies. And men I've seen are commanders/prominent political figures, or guards. But why aren't there men in the colonies? Did they die in the war, or get enlisted as guards or Eyes? Did most of them manage to escape or have the ones stuck in Gilead who couldn't get out fill other labour/work positions that we just haven't seen yet like craftsmen, plumbers, gardeners etc.?