Most Bizarre Problem I've Faced

Alright, I've seen some weird problems in my short time in IT, but this one is really just the absolute most bizarre thing I've seen. I would like to start with saying that normally I would be the first person to be skeptical of my account of this tale, however I took rather detailed notes, so I'm fairly sure I didn't do anything to cause this.

To start off with, this could all have been avoided if I wasn't an idiot and hadn't forgotten to put the new servers in the backup jobs when I built them. This has also all occurred between around 3:30 PM CDT on Friday, 4/24 and the present.

I needed to do a Veeam restore on a virtual machine to pull some settings off of the old server that the new one was replacing. We'll call them $old_serv1 and $new_serv1. So I kicked off the restore jobs into a datastore that was previously occupied, but had been cleared out recently. We'll call it $DS2. I messed up the restore settings a few times, causing some restore failures. I finally got the settings right, but the restores failed again and I noticed that the error was actually saying that the .vmx file that was just restored was not accessible because the path did not exist. When I went to look in $DS2 there was nothing there, so I figured that there was something with the device backing. I logged into the Hitachi array and started looking around. Everything appeared normal according to that, so I decided to try recreating the LUN/datastore since there was nothing on it. I deleted the datastore and remove the LUN from the host group (Fibre Channel mappings) and left it for a second to make sure nothing important went offline. The LUN mappings were 1 -> 1, 2 -> 2, etc., but you can never be to careful with storage and LUN mappings.

Go to the bathroom, come back, oh sh**, one of the other LUNs is offline. I panic and add the LUN back to the host group mapping, but nothing happens, the datastore doesn't re-up, nothing. I start searching KBs on what to do. Ironically, the datastore that went missing, $DS1, is the datastore that held $new_serv1 and it's counter part, $new_serv2. Luckily this place doesn't care if stuff goes down after closing time, no one does any work after hours. I send a quick email to the contact letting them know what happened and I engage VMware support. After playing the "how long can it possibly take to open a support case" game I get on the phone with a tech, and we start going through the troubleshooting process. Long story short here is that we do a hexdump on the two devices in question, and the device labeled $DS1 is signed as the device backing $DS2, and the device backing $DS2 seems to have been formatted because there is no file system signature. At this point VMware says contact your storage vendor, this isn't our problem.

Funnily enough, I've never opened up a case with Hitachi before, their stuff just sits in the corner and runs, and if it doesn't, I have backups any way. I wasn't quite sure what to expect, but what ended up happening was that I was playing phone tag with various people gathering log information until 4 AM, to then be told that they wanted to bring in a VMware expert, but he wouldn't be available until 12 PM. I say fine, and go to sleep, manage to actually get about 8 hours, despite how terrible all of this has been.

After hanging up with Hitachi, before going to sleep, I try a quick trick that might help the situation. I was worried that it might also destroy any data that I may have been able to recover, but I was willing to try anything to get it to work. I went into the Hitachi, removed the devices and re-added them with different H-LUNs (host LUNs, basically the LUN number that is shown to the connected system). This allowed me to successfully recover $DS2, but it was recovered from the device that should contain $DS1, and there was also nothing in it (to be expected, I deleted everything any way) and only had about 50GBs of space available (not to be expected). Decided I had best wait for Hitachi to take a look.

After an uneventful conversation with Hitachi that lead to suggesting that I call VMware back and see if they can repair the file system signatures, I did so. I also requested that we do a conference call with Hitachi regarding the issue, and let them grill each other about the problem. Essentially Hitachi forced VMware's hand, and they finally explained that in the logs what they were seeing was that when the device backing $DS2 went offline, it looked like the device backing $DS1 went off-line as well, and then got formatted when it came back in. No one could explain why this had happened, or how it would really even be possible. VMware's response to this was to essentially blame me and say that I had formatted the datastore by accident. Finished that conversation at around 11 PM, and just got finished rebuilding the servers a few minutes ago, just waiting for the restore jobs to complete.

Anyone seen anything like that before? I can't even begin to imagine why a server would drop a LUN when another one goes missing, and then when the other one comes back in, I guess it assumed that it was the other one, even though I had just deleted the datastore on it, and signed it as the old one.

TL;DR idiot forgets to add new VMs to backup jobs and storage does some weird stuff causing data loss and forcing said idiot to rebuild servers from scratch again.

Any way, wanted to get that off my chest. Back up your shit. 26 hours listed on the ticket, 24 hours that I didn't need to waste on this.