Just a data point on SMART value 22 (Helium_Level)

Discussion

I have a 10x8TB zpool filled with WD easystore red/white disks.

All the disks are almost 5 years old now (disks have around 40,000-41,000 hours) and the pool is about 80% full.

For the past year I have noticed the Helium_Level attribute decreasing on one of the disks.

About 6 months ago it crossed the threshold of 25 (25% since it started at 100?) where it is considered FAILING_NOW.

I have been continuing to use it in the pool daily since then and currently it is down to a value of 7. ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 22 Helium_Level 0x0023 007 007 025 Pre-fail Always FAILING_NOW 13

The disk is still seeming to operate like normal despite having been below the 25 threshold for about 6 months so far and under 10 for several weeks now even.

I have seen 0 errors on the disk in the zpool status or anything in the kernel logs.

I also do monthly scrubs so this disk has been scrubbed more than a few times well under 25 Helium_Level.

I will continue to monitor and use the disk until it actually shows signs of data failure in the zpool.

Just thought you guys may find this info interesting or useful.

EDIT NOTES ADDED

All other 9 disks are still showing 100/100 Helium_Level.

Temps have historically been in the 30-40C range and the disk is still its normal temperature.

Posted Aug 10, 2022, 4:35 PM UTC · edited

4926
HTWingNut1TB = 0.909495TiB•

Good info. Was wondering if/when we'd hear about any helium issues. Seems it's a non-issue though even with a significant decrease in that value. It could possibly be a faulty sensor too?

15
metropolis_pt2•

Afaik there is no dedicated sensor, the drive firmware is monitoring the motor current which indicates the drag of the platters in whichever gas mixture it is spinning in. As helium is dissipating the drag on the platters and thus the current increases.

26
HTWingNut1TB = 0.909495TiB•

I figured they'd come up with something clever. Just curious how accurate it is and/or what happens to OP's disk as the helium counter gets to zero.

6
SirMasterOP64TB RAIDZ2 + 64TB RAIDZ2 backup•

I found this:

https://forum.hddguru.com/viewtopic.php?f=1&t=35770

And mentioning this:

Method to detect helium leakage from a disk drive (US patent #434987): http://patentimages.storage.googleapis.com/pdfs/US7434987.pdf

8
HTWingNut1TB = 0.909495TiB•

Great info! Thanks!

2
greywolfau•

Could this result in a sudden failure rather drive errors slowly mounting?

Like a burnt out or overloaded motor?

2
clickwir4TB•

Absolutely

Remember, backups are backups. Sync, copy, and RAID are not backups.

2
SirMasterOP64TB RAIDZ2 + 64TB RAIDZ2 backup•

It could possibly be a faulty sensor too?

Maybe, but not sure I would expect it to intermittently but consistently decrease. But possible that is a failed sensor behavior as well.

5
HTWingNut1TB = 0.909495TiB•

Are the other ones still showing 100?

2
SirMasterOP64TB RAIDZ2 + 64TB RAIDZ2 backup•

Yes they are. Good question, I will add that.

2
HTWingNut1TB = 0.909495TiB•

How have temps been? Maybe you haven't monitored that detail over time, but I'd be curious maybe how it's running compared to the other disks at least? Sorry for so many questions. Just am interested in failure mode if something does happen. If it does fail, it will likely be pretty catastrophic as the helium escapes and head can't maintain it's minimum fly height will likely crash with the disc. But who knows. Thanks for sharing.

1
ifthenelse196KiB•

I wonder what will happen if the drive is stopped, cooled, and restarted in the absence of helium.

1
HTWingNut1TB = 0.909495TiB•

From my understanding and articles I've read is that the flying height of the head is calibrated to helium and loss of helium will result in a head crash in pretty short order. I'm sure a startup would propagate that catastrophe pretty quickly. Although they never said how much helium could be lost before running into issue. Considering OP's hard drive has lot over 80% according to the sensors, it's promising if it keeps running regardless.

2
msg7086•

What you want to monitor is what would happen when helium level drops to zero. The PCB/firmware may refuse to spin up the motor if helium level is critical.

We also don't know what helium level = 0 means. It's possible that even with helium level 0 there's still plenty amount of helium sitting inside and it would work just fine. (Just you know that helium is not really needed to run those drives, as long as the firmware doesn't stop the motor from spinning.)

8
SirMasterOP64TB RAIDZ2 + 64TB RAIDZ2 backup•

Yeah, I think I will make another post when the disk does finally fail, and describe the info then as well as link back to this post.

Personally I didn't think low helium would cause the drive to immediately fail. It will be interesting to see how long it lasts being such a low value now, but it will still only be 1 data point.

12
IHaveTeaForDinner•

Did it fail?

1
SirMasterOP64TB RAIDZ2 + 64TB RAIDZ2 backup•

Nope, no errors yet. Running 24/7 in my lightly loaded ZFS pool.

Been at 1/100 helium level in SMART for s few months now.

1
IHaveTeaForDinner•

Remind me! 6 months

1
RemindMeBot•

I will be messaging you in 6 months on 2023-02-11 07:17:23 UTC to remind you of this link

1 OTHERS CLICKED THIS LINK to send a PM to also be reminded and to reduce spam.

^(Parent commenter can ) ^(delete this message to hide from others.)


^(Info)^(Custom)^(Your Reminders)^(Feedback)
1
SirMasterOP64TB RAIDZ2 + 64TB RAIDZ2 backup•

The helium level is now a value of 1, and raw value 1.

3 months ago when I posted this it was at a value of 7 and raw value 13.

Disk is still running perfectly fine so far.

1
msg7086•

Thanks for sharing!

1
clickwir4TB•

Yes helium is needed for normal operations. They wouldn't go through the trouble of using it if it wasn't.

1
msg7086•

It's up to the firmware. When that check is bypassed, drives CAN work under regular air condition and you are able to recover data from it. This was verified by data recovery experts.

2
gargravarr211240+TB ZFS intermediate•

Just had a look at my Exos X12s which are also helium, and I can't see the attribute on the SAS drives even with smartctl -x. Anyone know how to see this on SAS?

4
SirMasterOP64TB RAIDZ2 + 64TB RAIDZ2 backup•

I know that attribute 22 showed up first when WD first introduced Helium disks.

I wonder maybe Seagate uses a different attribute?

3
gargravarr211240+TB ZFS intermediate•

SCSI-based drives report a lot differently to ATA-based ones.

4
Glix_1H•

Thanks for this.

What is a rough average value for your other disks? I’ll have to look at mine tonight and see where they are at, mine are roughly 3-4 years old

1
SirMasterOP64TB RAIDZ2 + 64TB RAIDZ2 backup•

The Helium_Level on all 9 other disks are all still 100/100.

1
malventano3.6PB Raw, 700TB HDD/40TB SSD Usable, Rest = Chia•

Data point for you folks: I'm reporting SMART value 22 = 25 for several hundred HGST/WD drives I have running here, so that appears to be the starting value / baseline (not 100).

1
pociej•

Remind me! 1 year

1
SirMasterOP64TB RAIDZ2 + 64TB RAIDZ2 backup•

5 months in, disk still working perfectly fine in my 24/7 ZFS pool.

Has been down to 1/100 for helium level for a couple months now.

2
pociej•

Thank you for answer.
I found this topic interesting and wanted to follow it in longer perspective.
My own knowledge is coming only from 2x WD100EFAX, spining 24/7 in Synology NAS for 30k hours as of today.
Their Helium level is still at 100 for both.
I'm very curious how your disks will behave and how long will last, that's why I placed this reminder.

1