Unplayable lag for me

I haven't found someone else having a similar issue but almost every second game is unplayable for me due to lag issues.

The weird thing is, there is no lag at the start of the game but after the first fights begin (cannon, swarm, etc.) it begins to lag. It gets to the point, were in late game the game completely freezes and I eventually die because I am not able to repair/rebuild. The other players seem to be unaffected, so I am guessing this is on my end.

My connection is fine tho... I have stable 20 Mbit/s down and 9 Mbit/s up.

Thank you in advance for your help!