Source
r/homelab•
WinOSXBuntu

PSA: Don't Buy Used Hard Drives

Discussion

TL;DR Don't buy used drives, they have no warranty and they will be five minutes away from palming St. Peter a fiver to get through the gates.

I have eight HP SAS 146 GB hard drives in my DL380 G7 and I thought I got a great deal on them because they were £12.85 on eBay.

Got the eight drives, the day after I got the server, one drive died the very next day, so I replaced it with the one SAS drive that came in the server, talked to the eBay seller to get a replacement sent out, they sent two and then a couple of weeks later, another drive starts making a screeching sound before it finally dies, replace the drive with another spare.

All is well for a few months until 02:00 GMT this morning, I am woke up to a phone call, quickly dismiss the call so I can go back to sleep, only to hear the telltale sound of a hard drive dying.

The click of death.

Had one last spare SAS drive to replace it with, pulled the failed drive, plugged the replacement in and thought "The RAID 50 will rebuild overnight" and I went back to sleep.

No, it didn't. Well it did, but no data was left, I only just found out this morning when my phone was using 4G instead of WiFi and when switching back over to WiFi, being greeted with no Internet connectivity, try my router VMs web page, no response, jump onto the ESXi web client and there it is, all 5 VMs, gone.

vCenter? Gone, OPNsense? Gone, the two Windows Server 2016 template VMs, I spent days working on? Gone, the budding CentOS template VM that I started working on the other day? FUCKING GONE! And just for kicks, orange lights have appeared on the remaining 7 drives. And to replace them with even SATA hard drives from Amazon? 35.40 for 500GB, nearly £270 for eight drives.

Do I have that money? Do I fuck. I don't even have the money for one. I'm in a dead end job with no end in sight, I've got an interview tomorrow but no guarantee I'll get the job.

I apologise for the language but as you can tell, I am beyond stressed. I used to have hard drives that had the chance to outlive me, used to be able to get 1TB for a tenner, not anymore.

Posted Jul 2, 2018, 9:03 PM UTC

126
jhereg10R310 PFSense | R710 ESXi | Cisco 2821 | UnRAID Docker / VMs•

I sympathize. But must state the following:

Raid is not a backup. Raid is not a backup. Raid is not a backup.

Where the hell are your backups!?!?

46
shysmiles•

Yeah, not much data either. Could have backed up the entire set on one 1tb drive.

10
Tourman36•

the PSA here should definitely have been "Moar backups!" not "Use RAID5 with shady ebay drives and hope for the best"

10
ImaCallItLikeISeeIt•

Exactly. I have 8 160 GB WD laptop drives in my r710 in RAID 5 and they have been performing just fine for months. Got them off of ebay for 80$ and (obviously) any data that’s on there is already backed up twice.

1
CanadasMooseOverlord•

Backup. A 6-Letter word beginning with a B, that prevents a 6-letter word beginning with an F.

8
Protocol73home.bellevuelabs.net•

!redditsilver

2
bmc2•

Lesson is don't buy super old drives, not that you shouldn't buy used drives.

I have a mixture of used and new drives in my array. Zero problems.

I also have a stack of 146gb 15k SAS drives. If you were in the US, I'd offer to mail them to you, but chances are postage to the UK is more than the drives are worth.

11
CptCmdrAwesome•

Yeah I agree, picked up three Hitachi 2TB SATA enterprise drives off eBay and had no trouble whatsoever, but I tested the shit out of them first, use ZFS, and backup the important stuff.

1
fmillion•

The actual PSA here should be: treat ALL drives as if they could fail at any time, regardless of how old or new they are - because they can.

I've had plenty of brand new drives that fail within days or weeks. A warranty will replace it for me, but I still could lose data if I don't have backups.

I've also got drives with 10 years of run time on them that still run fine. Do I treat them as "on the brink of death?" Absolutely, but I do that for ALL drives, since any drive can fail at any time, and at least my own experience has been total catastrophic drive failures. I've had many many more mechanical failures than I've had "bad sectors creeping up" failures.

But all that said, I'm not sure if you got the 15K SAS drives, but the 15K 146GB HP drives are usually rebadged Seagates, maybe Hitachi's. The point is, 15K drives run HOT. You need adequate cooling to keep them going reliably, especially in large groups. A single drive can probably run in a desktop PC case without causing too many issues, but cram too many of them into a small space and you can easily have some heat issues. Even more important to us homelabbers, since we are not typically running our servers in tightly climate controlled spaces - I shudder when someone says "I just moved my lab to the attic - it's 100 degrees F in there during the summer and the fans are loud but everything seems to be working!" Heat is the enemy of the mechanical hard drive, and 15K drives generate a lot more of it than typical consumer drives or even nearline SAS drives.

Of course I understand the plight of the "poor homelabber" - I'm sure you WOULD buy a tape drive or multiple external drives or whatever if you could, but if you can't, plan appropriately - if you have 8 drives, use maybe 5 of them online in a RAID6 array and then do OFFLINE backups to the other three - periodically copy all the data to them and POWER THEM OFF. Drives last far, far longer in cold storage than in online use. It's fun to go "I did a RAID0 on eight drives and LOOK HOW BIG MY VOLUME IS" and it's also easy to feel like you're "wasting" the drives that are being used for redundancy, but consider it an insurance policy for your data.

(Example: I have 5 WD 8tb Reds. I run them in RAIDz2, so I have 24TB of usable space. Wouldn't it be AWESOME to see 40TB? Sure, but then I'd have no redundancy at all, nor can I back up that much data offline anyway (I can't even backup 24tb, maybe about 15tb across all my offline drives, but I ignore things like podcast mirrors that I can at least likely rebuild if needed)

7
SIN3R6YMarriage is temporary, home lab is for life.•

I buy used drives all the time, sure some die in short order, but in general i'd say i have about a 95% success rate when it comes to lasting at least a couple years. I've had new drives fail in short order before too. Moral of the story is have a good redundancy and backup strategy.

3
stone-sfwballer on a budget | MacPro-5,1+ESXi-6.5•

my first step for any new drive is check them with a smart status app like DriveDX (mac), immediately return any that fail.

6
GoingOffRoading•

"Do I have the money? Do I fuck?". I'm sorry, but I'm stealing that

6
waterbed87•

Don't blame the drives honestly, blame yourself and learn from the experience. Running one of the riskiest RAID configurations (outside of 0) on used hardware with no backups is setting yourself up for data loss.

Consider RAID 10/60/6 in the future as they are more resilient and treat an array as you would a single disk. It needs to be backed up if you care about that data. A RAID is more redundant than a single drive but it is absolutely not a backup strategy. Have data you care about? 3-2-1 Backup. 3 copies of your data, 2 locally, 1 offsite.

5
hyperactivedog•

Raid 50 + used hardware... hahhahaahaha.

https://www.youtube.com/watch?v=gSrnXgAmK8k <- this is your life...

RAID10, have backups on top of that.

Every array I use is either a single drive, RAID1 or RAID10. RAID5 is fragile.

4
packetheavy•

This is probably my favorite Linus video.

2
waterbed87•

That guy could really use just one dedicated system administrator. He's not an idiot but a man of best practices he is not.

5
ImaCallItLikeISeeIt•

I knew what this was before I clicked.

2
elforesto•

That really sucks. I once lost an entire HDD of music because a kid found the joy of rapidly switching the PC on and off. Definitely was kicking myself for not having backups.

Always remember that RAID is about uptime while you recover from a drive failure. This is why a lot of important arrays are "belt and suspenders" nested RAID (e.g. parity + mirror): you can tolerate multiple failures both before and during a rebuild. One of the nasty secrets of hard drives is that if they're from the same run, they're pretty likely to fail around the same time, and once you get past about 3-5 years (depending on the drive) you're going to be living on borrowed time.

But it's still not a backup. It to prevent you from having to restore from backup while you replace one or more failed drives, all the while praying that you don't have additional drives fail while the stress-inducing rebuild dares another one to die. In theory, you will replace drives faster than they will fail. In practice, you'll eventually draw the short straw and lose the whole array.

4
ItWasDNS•

Damn son... I just threw away 20+ of the Dell branded 146GB SAS drives. Too old to be trusted.

3
ava1ar•

Who uses 146Gb Hard drives today? They are way to small to store anything for whatever price you might get them. Not mentioning they are hot and noisy.

2
itomeshi•

I've had my small raid-5 arrays die too, and I feel ya: I'm sick of it.

Along with the 'raid is not a backup' rule, I think we need a new one:

RAID parity isn't worth it.

I've given up on it. Currently building a RAID 1+0 set, and since I'm using ZFS, while I'm starting with a 2 way mirror, I may pump that up to a three way mirror. RAID-5 is just too FRAGILE. Seriously, notoriously, absurdly fragile. I don't have the patience for it anymore. Re-silvering via parity from a disk loss is such a slow and risky process that it's not worth it. I want to just get the ZFS pool running so I can transition from 'OK, yeah, I have a backup' to 'I have reliable, distributed, encrypted snapshot backups in the cloud, with automatic rotation'.

I imagine you are running on a big iron server, but can I recommend something like Ceph? Ceph's more appropriate for Docker, but I'm sure there is some sort of VMWare-like equivalent.

BTW, good luck in the job interview. Don't let this stress you out. This is fixable.

2
aaronduce•

Thought of just doing a raid 0 and having a backup?

1
TurbulentEscape•

I have 43 SCSI disks (9, 18, 36, 72 & 146GB varieties) for my old HP MSA 1000, & 2 DL360G3s. Of the 7 years I ran most of them continuously, I've had 2 fail. One after a power failure and one randomly started failing.

All these were bought used probably a decade ago. I know for a fact that 18 are still functioning properly, but those machines are no longer running because they eat power like nobody's business.

2
redeuxx•

Thanks for sharing, but your experience is just that, your experience. Used drives are fine if you know that it's probably more likely to fail than new drives and to take steps to make sure you don't lose data. Also research your seller. I had one drive go bad within two weeks and I was just gotta write it off as me being unlucky. I put in negative feedback on eBay and the seller cared so much about his feedback that he sent me a new 8TB WD Red Pro to replace the used 6TB Red. My experience is that less than 10% of the refurbished drives I've ever bought have gone bad, and they go bad quick. This goes back to researching the seller and what their return policies are. The others have stayed alive for as long as my drives bought new ... one nearly has 50k hours on it.

2
g-a-c•

I feel for you, and I want to hand you the stack of 15k RPM SAS drives on my desk that I have no need for - but I haven't been able to DBAN them all to remove company data, and really I'd just be perpetuating the issue, since I can in no way guarantee their condition...

1
AirsoftinAction•

On the flip side of this, I have 7 very used HP SAS 146gb HDDs in my DL380 G7 and have yet to have a single failure. In the event I do, I also have a hot spare on standby and a few extras I can quickly swap in.

1
pdhcentral•

I just got 1 used 8TB & 1x4TB disk. Both have less than 200 hours on, show no SMART errors and surface scans are fine. Good enough for me. I have backups. The price is the best part, only £130 instead of £300!

1