How much do the legion buffs matter?

Just wondering because im a Night Lord at heart and I want to paint little lightning bolts on all my guys. I know that Imperial Fist rocket spam is quite oppressive and generally frowned upon at the upper levels. But like let's say im just doing a casual game how much is the buff my opponent gets for playing a more rounded legion vs my niche buff going to make a difference?