Fire in the server room!

One of my more memorable times from being a sysadmin was when we had a fire in our server room. This was a few years back, when I worked at an SMB with around 300 employees. Our on prem server room was one we had built out when we moved into the building a couple years earlier.

It had originated as small server room/telco room, which we expanded into a small office next to it. We made it large enough to house around 5-6 racks in one row and had it all nicely wired up by our favorite contractors. No raised floor (one story building on a slab), only using commercial building AC (this became a different story - standard AC units aren't designed to run in below freezing temps outside, but servers still need cooling).

We also inherited a 22kva Leibert UPS, one of the always on varieties so it converts utility power to clean power. I believe these are called in-line UPSs. We also had an ATS (automatic transfer switch) connected to our generator. The ATS and non-server room power panels were all in another utility closet around 100' from the server room. We were able to get a server contract in place for the UPS and they regularly came out and serviced it, replacing batteries as needed and other PM if needed.

Since this UPS wasn't big enough to handle everything we had a few smaller, rack mount type UPSs to take up some of the extra load and give redundant UPS to some of the equipment.

One nice summer day I am on my commute home, almost home, and get a phone call from my manager, $M, that I needed to come back in. There was a fire in the server room! He had stayed a little late so happened to be there when it happened.

I got back and the fire department had a big truck outside the front door and a few other trucks and a bunch of fire fighters, $FF, wandering around. I was glad to see there wasn't a big jet of flame coming out the roof of the building or black smoke billowing around. I found $M outside and we were able get permission from the $FF to go into the building back to the server room area. They were still working on the 'issue'.

$M and I got back to the server room area and we found out that the problem was with the tall, grey, piece of technology - it was smoking and jets of fire were coming out the back of it. They had sprayed it with an extinguisher which they brought with them which contained water and dispensed it in a fine mist. The $FF said this worked well on electrical type fires. It had put the fire out but the $FF wanted to get electric cut to this device since it was hard wired into the utilities with a thick cable. Of course, this tall, grey, piece of technology was the UPS and that thick cable was its power feed from the ATS in the utility closet.

One may wonder why no fire suppression inside the room came on. This was because the fire had never left the UPS box and didn't heat up the room enough to cause the water (yes, water) sprinklers to kick in. We had never had the budget to convert the room to halon. Halon or water really would not have made a big difference in this case as the issue was that until power was cut to the device that it still would have been dangerous. Water could have made a nice shock hazard though.

Anyways, we all went to the utility closet and had to get someone from the building management company since they had the key to this door. They showed up, we got in there, and turned off power at the ATS and the other panels which lead to the server room for power not going through the ATS/UPS setup.

At this point the $FF start to go into cleanup mode and we talk a bit about our next steps. They eventually leave and it's now me, $M, and our other sysadmin, who also came back in to help out. I think we had a few other non IT and big boss, $BB, came in to 'help'. The three of us talked about it a bit and we wanted to get systems back online as much as possible. To do this we needed power to the server room. To get that we needed to bypass the UPS so we had to call our electrician to come out after hours, probably around 8pm or so by now, to do some emergency work.

Due to the $FF usage of the water mist extinguisher we didn't have any other pieces of equipment fail or get destroyed due to water inundation. When the electrician, $E, got there we asked if we could turn on the other panel while he worked on giving us a non-UPSed set of twist-lock plugs for all the power strips going to the server racks. We got the other panel back on and started deciding priorities for what needed power to get the essentials of the servers and network back up. Most of the networking was on the smaller UPSs already but we were able to move some of the server rack power strips over. We wanted to get the SAN going again first and then a few ESX server up in the cluster to handle the most important VMs.

We wound up having a big chunk of the servers running on utility power for a while, a scary thing in the US during the summer - frequent thunderstorms can cause power flickers or high AC load on a really hot afternoon can cause a brownout, all of which we had no protection from for the foreseeable future.

We were able to do all this that evening, I think we stayed until around 2 or 3am getting this stuff running knowing that we would need to come in the next morning at the regular time to handle our normal 'fires'. It was fun calling our Leibert rep telling them about our incident. They had never heard of such a thing happening before and replaced the entire UPS as part of the service contract. I think we paid around $7k/year for that contract, and we made our money's worth on it!

It did take several weeks to get the replacement unit in and wired up and then we had to plan on how to cut back over to the new UPS since we didn't want to overload something by unplugging one of the redundant power supplies, which causes the other power supply to take full load and since the small UPS and/or panel was a band aid solution it wasn't designed for full load. There were of course several pieces of equipment which had only one power supply, like older Cisco routers, which we had to plan a down-time for.

What an ordeal!

So, N+1 in everything, even that expensive UPS which should have N+1 inside it, is important and can make a difference.

P.S. - we still didn't get a halon system after that. We got a dry fire extinguisher in the server room and put shields under the sprinkler heads on top of the server racks to hopefully direct water away from the servers in case of a future event.

TL;DR - old UPS in the server room caught fire, fire fighters killed power to everything, emergency call to electrician to get some stuff back online.