Question on the Downfall of Viltrumites

Forgive me I've only ever seen the series and not read the comic books. However I am about halfway through the new season and confused on something.

So the Viltrumites we're essentially brought down to a population of only 50 people due to a disease that was exposed on their home planet. However, they are a well-established empire in full conquest mode, shouldn't they have theoretically of thousands if not hundreds of thousands or even possibly millions of troops all over the universe?

For example all the Romans were not condensed within Rome, a good portion of them were spread amongst the territories that they had conquered along with tens of thousands of troops that were deployed for different campaigns.

Wouldn't there be far more Viltrumites left alive when you consider the expanse of the universe and the necessity to keep soldier stationed across different planets to ensure their rule remained unchallenged and consistent? Were they also exposed to the disease at the same time? When they discovered the extent of the disease and how devastating it was wouldn't they give them commands to stay stationed at their own planets to preserve the empire and avoid further casualties to healthy Viltrumites?