Source
r/sysadmin•
effgee

Friendly reminder.. Don't trust SMART, or Samsung Magician...

Just a little rant, bought a Samsung 870 EVO 4TB about 6 months ago, have barely used it at all. Written only about 1.6 TB of data in total to it (before the following tests).

Started getting high cpu usage (Windows 10 ) for the system interrupts process and noticed a ton of events in Event Viewer with bad sector found etc etc.

These errors started appearing after I started backing up (Veeam) the drive.. (of course they did...) which previously held only some replaceable data but as I started using it for some more important things, it was time to back it up.

What is really not great is that the windows built in tooling for disk health & the performance monitor reporting, Crystal Disk Info & Samsung Magician all give the drive a "healthy" rating. sigh

  • Both the short and extended Samsung Magician SMART test both failed to complete.

  • Also ran the Samsung Magician diagnostic scan "Full Scan" mode, which showed a smattering of bad lbas, and gave me a "repair" option.

  • After repairing I ran it again and it gave me more (but different) bad lbas.

  • Repair appeared, and used. Ran the Full Scan again, and the same LBAs are still considered bad.

  • Initiated warranty return.. but I continued doing some testing.

  • Filled the drive completely using https://panthema.net/2013/disk-filltest/ .Drive filling and verification worked without error.. hrm.

  • Re-ran Samsung Magician Short + Extended SMART test, which now completed. (wut)

  • Re-ran Samsung Magician Diagnostic Scan "Full Scan", which completed without showing any bad LBAs. (wut wut)

  • Drive still shows as "HEALTHY" in all diagnostics.

Final SMART statistics look like this.

https://imgur.com/a/AjPWNkZ (few other images as well...)

Sigh. Just a friendly reminder, don't trust SMART... or Samsung Magician (Or NTFS.. I wonder how stable ZFS on windows is at this point.. hmm).

Posted Jan 12, 2022, 5:52 PM UTC

1221
jpc0za•

Two critical alerts in SMART but it shouldn't be trusted?

I'd warranty return the drive

23
effgeeOPTechnically Manager•

RMA Process has been started already..

I guess I mean more the "SMART" evaluations from the OS/Manufacturer software.

I mean cmon, exactly what you said, two critical alerts and the software says "drive is okey dokey"

It is not.

3
polypolymanJack of All Trades•

When I monitor SMART, I take a look specifically for values 187,188,198 and 199 (they all won't be on every drive). If any of these values aren't 0, it's a good chance the drive is on its last legs. Looks like your 187 is spiking like crazy.

EDIT: of course, forgot to mention: make sure you're looking at "raw values", not the other columns. I've never seen those other columns actually mean anything.

18
MyWorkIsNotYetDoneGovernment IT Stooge•

Thanks for the info. I saved this comment for future SMART scans.

2
Sunstealer73•

Check out Stablebit Scanner. It's great for continuously testing and monitoring drives. If you also use their DrivePool tool, it will empty out a failing drive automatically.

3
effgeeOPTechnically Manager•

That is interesting. Heard of drivepool but not the scanner part of it.

1
hammong•

Ran across a very similar situation on my 980 Pro 2TB on a ASUS Z790-E motherboard, 11900K. After a few months of light use, I noticed that my image backups were failing on the system. File-by-file reads never failed, no visible performance or integrity issues, SFC /scannow never picked up any corruption, etc. By all accounts, everything "worked" fine.

​

But, I checked Windows Event Viewer and saw many bad block errors in the System Log. Ran Samsung Magician (which was a month or two out of date at the time) and it would not complete the Short or Long SMART tests. Short Scan would work,. but Full Scan would show many red blocks randomly in the map.

Updated Magician, flashed the latest firmware, updated Z590-E Strix BIOS, and re-ran the tests. Same results, and more bad blocks on the map (different ones, visually).

So, then was reading where some people have done a secure erase after updating the firmware and re-installing from scratch fixed the issue. I then backed up (file-by-file), did a secure erase in BIOS (this is a triggered hardware secure erase on the drive, not a block-write erase), install a fresh copy of Windows 11, latest and greatest drivers for Z590-E.

Result? Short and Long SMART tests pass, Short and Long Scans pass with ZERO bad blocks listed, no errors at all in Windows System Log. I've run the test suite weekly for the past three weeks, and no new bad blocks have appeared.

TL;DR - I'm chalking this up as either bad firmware on the SSD or bad firmware on the motherboard BIOS causing this. My SMART diagnostics in Magician still show many TENS of thousands of media errors from before the secure erase, but the number is not increasing.

YMMV but I am not convinced the drive itself is bad, but it might be a bad combination of OS, BIOS, drive firmware, and drivers at work.

1
wolffcaleb•

I'm having this same exact thing happening with a 3 month old 2TB 980 Pro and a B550-F and a Ryzen 7 5800X.

​

SMART tests won't run, long scan shows like 30% bad and recovery prompt fails, short scan looks 100% good but says something along the lines of "defects found on drive please check help" - help says it should show a recovery option, but it doesn't.

​

Event Viewer displays bad block issues. Dozens of them.

2
hammong•

Same exact symptoms! Back it up, flash the SSD and System firmware, secure erase, re-install and make sure you got the latest drivers and see if problem recurs. It's been > month now since I did mine, and zero problems since then.

I'm firmly convinced this is a bad older firmware, but not 100% sure if it's the motherboard or SSS firmware that's the fault.... or both.

2
Ohmec•

Holy crap this is literally EXACTLY what's happening to me! I have been so bummed today because my drive seems like it's failing and I spent $450 on it. I got it after my first gen Corsair MP600 died after 2 years, because I have always trusted Samsung flash quality.

I hate to ask you to elaborate further, but can you give me an exact, step by step of how you recovered the drive? How did you back it up, what firmware did you flash on it (mine was just updated to latest firmware, is that a problem?), I can do the secure erase, and then when you say "re-install" do you mean re-copy the backed up drive image back to the drive? Or just reinstall a fresh OS copy?

For context, I haven't even been able to complete a full scan in magician with this drive because it is taking over 8-10 hours to even get half way.

1
hammong•

I did not "recover" the drive or any data on it. I keep my critical data on a server in my garage that is fault-tolerant, and then I periodically replicate my "really" important stuff off to private cloud storage. So, data loss is not an issue for me.

The steps you listed are roughly it. Backup (if necessary) the data, update the Firmware to the latest version (I used the Samsung bootable USB disk image to do this), and then Secure Erase the SSD using either the Samsung bootable USB or the BIOS secure erase utility if your motherboard has one.

By "re-install" I did just that. After flash/secure erase that drive is completely and totally blank - no partitions, no files, no OS etc. I popped in a Windows 11 bootable USB, installed Windows from scratch, installed the required drivers and Samsung Magician, and here we are months later with no errors at all.

2
wolffcaleb•

RMA on the drive resulted in them sending me a new one. When I pulled the old one out of my PC it actually look as if something clear and liquid was leaking out of it in a thin layer. No sure what that was all about.

1
Awakened_light•

what firmware was on 870 evo?

1
effgeeOPTechnically Manager•

Unfortunately the drive is long since Rma'd. I dont think i have a screenshot of the firmwares version.

If i find a screenshot i will post it.

The errors started and there was a firmware update which I did update it but it continued to degrade.

1
ime1em•

s 187,188,198 and 199 (they all won't be on every drive). If any of these values aren't 0, it's a good chance the drive is on its last legs. Looks like your 187 is spiking like crazy.

looked like you were having the same issue as me and other 870 owners. RMAed my drive last week. Going to run a full scan just in case.

​

https://www.techpowerup.com/forums/threads/samsung-870-evo-beware-certain-batches-prone-to-failure.291504/

2
effgeeOPTechnically Manager•

Sure looks almost exactly like what I encountered! Thanks for the link. Hope my new one doesnt do the same thing.

1
JetsNovocastrian•

Also gonna say thanks for the link! I'm in the same boat with the 4TB drive. Bought the hard drive a while ago, so not sure if I'm still in the RMA window. Shame, because it cost an arm and a leg at the time!

1
ime1em•

U should have a 5 year Warranty so it's good. Unless u passed the TBW written.

1
bbqwatermelon•

Dont trust SMART for SSD you mean. There is no way to reliably predict when SSDs will crap out. They just fail one day or outlive their TBW. SMART is only useful for statistics and temperature for SSD but I think it's bad advice to ignore SMART if using spinning rust...

-4
gamebrigada•

This is false. Complete drive failure is caused by either a failed controller or physical damage like overheated drive. All SSD's are overprovisioned, once a cell stops being reliable, the controller moves the data to a healthy cell that has never been used and adds an error count to the smart data. You'd be amazed at how many writes drives can handle, way outside of reach of normal use. Check out the SSD endurance experiments done by ServeTheHome and TechReport.

7
bbqwatermelon•

You basically reiterated what I stated with additional theoretical fluff. Out of curiosity how many catastrophic failures have you personally experienced where SMART gave any indication ahead of time with a SSD? Yes those experiments were of interest but really do not reflect real world service life at least of the particular drives I have handled.

0
_aD•

No way to predict SSD failure except when it's failing self-tests and showing a ton of bad sectors?

5
bbqwatermelon•

Did it die though?

1