Source
r/homelab•
miningdroid

Faulting disks with NetApp DS4243 + Proxmox

Help

I've got a proxmox server (HP DL360 G7) connected to a DS4243 (5 x 2 tb SATA, 2 PSU, 2 IOM3) and initially it was working great. After about a week I noticed one of the drives wasn't flashing, and it got kicked out of the ZFS pool! I removed and reinserted the drive, and it resilvered just fine. No issues for again another week, then a different drive went unavailable. This time I took a look and the console is full of messages like:

Nov  2 17:11:31 moya kernel: [360021.138211] sd 4:0:0:0: [sdb] tag#140 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Nov  2 17:11:31 moya kernel: [360021.138217] sd 4:0:0:0: [sdb] tag#140 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00

The second time it was a different disk, and in the process of moving my VMs off the pool, another disk failed out. Again, reinsert and the resilver took 6 seconds! I'm kinda at a loss here, nothing seems to be actually wrong here. I don't have the NetApp software so I can't interrogate the disk shelf to find out what's wrong. I've only got one QSFP cable, so I can't multipath it thru the second controller, but I've got another cable on order. I do have another SAS card (PMC Sierra) which I'll eventually try if this keep on happening, but I doubt it's the SAS card.

I copied all my VMs off the shelf, and then rebuilt the pool as RAID-Z2 because now I'm scared. Copied them all back and it's running well so far. Fingers crossed!

Any thoughts?

Posted Nov 5, 2019, 5:57 AM UTC

06
m0v_3r•

Did you try to reseat the IOM module?

2
miningdroidOP•

I didn’t, because it’s difficult to reach, but I could try that. If I get another drive out, I’ll do that.

1
Grandsinge•

So I had this happen a few months back. Turns out it was a bad controller (HBA) and I also had a bad drive.

2
miningdroidOP•

I see. Well I do have a spare HBA I could try. I kinda suspected this might be the case because one of the heat sinks moved on the card I’m using making a cracking sound as it did. I removed both heat sinks and the paste was all dried up. I re-pasted both chips and popped the heat sinks is back on. Do you think it would make a difference if I tried ports C or D, as opposed to the A port I’m using now?

1
Grandsinge•

You could try. But honestly HBAs are so cheap it isn't worth the headache imo. I tried troubleshooting cables, ioms, etc. And until I switched HBAs I would randomly gets dropped disks. The interesting part is a bad disk seems to have triggered it because the first disk that did drop out did have several bad sectors.

2
merkuron•

I’ve also seen bad disks cause upstream errors that “look like” controller errors; I’d replace the cheapest component first and work up from there.

2