Death by Mucus.

Medium

So I've got a server that's connected to a external drive array. The raid card this drive array connects to had a dead battery.

As a side note it also had some major drive issues. So the array is configured as a raid 5 with no hot spare. It was a trade off of some redundancy and less cost. There were some bad blocks on the drives. Then a separate drive completely died. Well since, for those bad blocks + dead drive, there was two failures which meant there is no recovering. Dell called it a puncture. There were three drives that were affected by these punctures out of 15. System Administrator showed them in a predicted failure state. The first tech incorrectly sent me replacement drives. I replaced them one by one and they slowly failed one by one. I called back and the next dude I spoke with told me the only way to fix this is to wipe the virtual disk and recreate it. He said the only way to be proactive about this kind of thing is run a consistency check in system administrator occasionally and as long as I get those bad blocks fixed before a drive fails then I won't have these punctures. That's not really pertinent to the story but I thought it was interesting none the less.

Anyways, I had to replace this perc battery. I had the top of the server open looking down in it removing the battery.

I'm going to take another route in this story to talk about the ac in the server room. A few years back we moved a lot of servers to virtual. That left a grossly overpowered ac unit blasting into a very small server room. So at the time it was about 56 degrees in the room. We've had trouble raising the temperature because it lowers the humidity. I know, it's counter-intuitive.

So, I had been in the server room for a little while at this point taking care of some other stuff. Then I was hovering over this server and all of a sudden I was made aware that a drip was forming on the tip of my nose. It was well within the specifications for possible dripage. I said a silent "Oh Shit!" in my head and slowly moved away from the server to take care of it. The server was off so I could have cleaned it up if need be but I was thinking if I never even noticed it I would have closed that sucker up and upon pressing the power button I probably would have heard a click and might have gotten some smoke.

Moral of the story, you better check yourself before you wreck yo'self.

Posted Oct 10, 2014, 3:44 AM UTC

163
PoglaTheGrateScript Kiddie and Code Ninja•

It's... kinda like reading a James Joyce novel

6
Rauffie"My Emails Are Slow"•

Heh, I thought I would be hearing the tale of someone projectile sneezing into a server and frying it. But this is ok.

1
MoneyTreeFiddy•

I thought the dell tech was gonna be the main focus of the story, but the nose really ran away with it.

1