Army desert

I got attacked and my whole army of variable amount of people deserted because they need building that supplies them on world map while they're being within city limits and I need to be over 350 people to unlock building I need when I'm being attacked while I barely scraped 100.
Am I missing something here?