TIFU by downing 3 production servers and almost causing the worst possible downtime the company ever had.
Phew. First time after work that I'm physically shaken up, since for one, that was a bad one, and for two, that was almost the worst one.
Let me start at the beginning: We are a software team, and we are currently working on replacing a bunch of old software. This old software goes ahead, looks at a server, figures out how much monitoring data it needs to read from a server and then reads that much data from the hard drive of the server.
We are re-implementing this behaviour in a software which integrates better with the surrounding software context, and which is a lot more efficient. Nothing too scary, one would think. The old solution never caused load on the hard drive because it was that slow and our new solution never really caused load on the hard drive either, I assumed because the small script on the monitored server was dog-slow.
But boy, oh boy, I was wrong. I was very, very wrong. Today I fixed some seemingly small threading issue and updated our test system and didn't really assume anything bad. Until the manager of the other project appeared at my teams place. Something already royally nuked 3 of their production servers for 10 minutes, and it appeared to go from one server to another.
At that point, my hands turned from normal to ice because I had seen in the logs that my system was doing something with each server for 10 minutes, which was a little bit faster than usual. Well, not just a little bit. A whole freaking lot faster. ... Oh fucking fuckety Fuckingsticks.
Ok, so someone sent a dev log outputs from our system when it started to work with a server and yup... exactly after our system figured "I'm working with server A", server A's disk utilization spiked to 100% and the production server and the database on that host just gave up doing anything. So the small script on the server wasn't dog-slow, it was lightning fast and backstabbed everything to get faster. No, I don't know why this wasn't ionice'd.
At that point I stopped the whole dang thing, removed every possibility to start the system again for today and just leaned back in my chair. The servers went back to normal. An outage-mail followed, I apologized publicly and told everyone why this issue won't happen again for now and more details would follow tomorrow.
But holy hell. If I hadn't been in the office when this thing went on a rampage, this would have easily been the worst night in the company. This software goes ahead and accesses all servers every few minutes. So it would have been a night where each server dies for 10 minutes every 30 - 60 minutes, and usually my system just disappears again quickly enough to not be detected because it's done and got all the data. And it's allowed to do so, because it ran for month without having any issues, so it had access to everything. And even worth, finding this culprit outside of someone asking me and me having the right hunch would be close to impossible because it goes in, stabs a server and gets out again, just like a monitoring ninja.
And even now, that I'm at home (with fresh VPN access to everything, my private cellphone and landline in posession of all emergency responders at the company and a beer)... I have a system which could completely kill our production environment for a non-trivial time, without being easy to fix at all. Holy hell. And it's designed to do this, sort of. I need to figure something out to not make this go wrong, because if this goes wrong badly now, it will cost me my job legitimately and it has potential to drag the companies products into bad places.
Posted Jul 1, 2014, 6:28 PM UTC
And this is why you never run something on production without testing it.
Good luck working on this though.