Weird data corruption on ZFS mirror; how to find the root cause?
I have two ZFS pools; two 4 TB disks in a mirror, and a separate 8 TB disk with no redundancy. The system only stores backups, and the backups on the 8 TB disk are less important, so the lack of redundancy isn't much of a problem.
A scrub runs on these pools once a month. Both in November and now in January (but not in December), the scrub found checksum errors on one of the 4 TB disks. 6 of them each time, and "768K repaired" each time. Back in November I figured I'd let it slide and look it up if it happened again. Since it did, I replaced the SATA cable for that disk, and also swapped SATA ports between the two disks, figuring that if the port/controller is broken, the checksum errors might start appearing on the other disk instead.
Immediately after doing this, when booting up again, the other, previously OK disk showed as UNAVAIL, and upon looking in fdisk/cfdisk, the partition table showed as DOS/MBR rather than GPT. I gave ZFS the full (zeroed out) disks on creation, so it certainly created GPTs on both.
Long story short, gdisk [sic] showed that the backup GPT was corrupted, and that the primary GPT contained some errors: the second partition ZFS creates after the main data partition (about 8 MB) was too big, and covered a tiny bit outside the disk. It also says the backup GPT was located outside the disk... which might be why it showed as corrupted.
The SMART numbers look fine for both disks; neither has any broken sectors, and I tested the disks thoroughly before I created the zpool. I also ran a long SMART self-test on the disk in November and it came back fine.
I copied the primary GPT to the secondary with gdisk's expert mode, and then recreated the tiny partition with the correct size (about 7.8 MB filled up the disk, vs the MB it was prior) with the same partition types and such, exported the pool, and imported it (zpool import -d /dev/disk/by-id pool), and it finally found both disks and started a scrub automatically.
However, literally a minute or two after starting the scrub, there were already checksum errors detected on both disks, 12 and 15 errors respectively.
It also found one file (in a snapshot) with permanent errors, which I find very weird. With just a few blocks corrupted per disk, how likely is it for those to occur in the same blocks of the same 50 MB file? I figure there must be a cause that could affect both at the same time, i.e. not the disks and not the SATA controller, but something earlier in the chain, so to speak.
Also, the corrupt file in question was created on December 5th, and the December scrub (with 0 errors!) was on the 10th, so the file cannot have been corrupted on writing; it must have been valid 5 days later. I can also be certain that the file hasn't been written to after December 5th, as the software in question only creates and deletes files; it never modifies them.
I ran a second scrub without changing anything, to see if it found the same errors or not (to see if the actual error was in the scrub result, e.g. when reading/calculating checksums), and the results were the same. 6 invalid blocks on both disks, causing the same file to be corrupted (including in all snapshots, since those are the same blocks).
Hardware/software versions:
This in on a rather ancient computer, but it's served me well for this simple almost-write-only use, until now.
Intel Pentium Dual-Core E2200
4 GB DDR2 RAM (non-ECC)
2x HGST HUS724040ALA640 4 TB 7200 rpm SATA disks (bought at the same time, serial numbers are close to each other)
1x WD WD80EZAZ (with no issues so far)
Ubuntu 22.04.3 LTS, fully updated
Linux 5.15.0-91-generic
zfs-2.1.5-1ubuntu6~22.04.2
Since the same blocks were corrupted on both disks in the mirror, I figure the error is most likely NOT due to bad disks or a bad SATA controller, since it would be extremely unlikely for such an issue to affect the exact same blocks on both disks.
I've been running stress tests for the CPU and RAM for several days now with no issues. prime95/mprime for about 15 hours ("blend" setting), y-cruncher for about 22 hours and counting, and 5 passes of "memtester" (Linux userspace software). I haven't been able to run a proper "offline" memory test like memtest86(+) yet as the computer is offsite, so I've only really tested 3/4 of the bits.
After that, I ran a simple "stress" test of writing data to the array and reading it back, letting ZFS checksum it. For part of the test I used random data, and for part I used /dev/zero and also manually verified that only zeroes were read back. No errors here either, in maybe 3-4 TB written+read.
So in short:
- One disk showed checksum errors in November and now, but not in December. There were no actual data errors; ZFS repaired them with data from the other disk.
- A different disk had a corrupted GPT and and BOTH disks now had checksum errors immediately after switching SATA cables and ports. This disk passed the scrub literally the day before.
- The new checksum errors happened such that the same 6 blocks of a file were corrupted/inaccessible despite the mirror, so they must have been corrupted on both disks (I assume?)
- The disk in the separate pool has so far been fine
- The corruption must have happened AFTER the data was written (since the December scrub found 0 errors 5 days later), despite there being no writes to modify that data post-creation
- A second scrub showed the same result, so the error doesn't seem to be in the scrub/verification, but on disk
Does the last point mean that the block checksums themselves are more likely to be what's corrupted?
Would such corruption be detected by ZFS? Is the checksum data also replicated? My understanding is that it is, and would be detected, so I'm at a loss as to what has actually happened here.
Also, regarding the GPT thing: it seems two or three ZFS disks have the secondary/backup GPT header in the wrong location, which is just bizarre. ZFS bug? I gave ZFS the whole disks, so it created those partition tables for me:
zpool history Ext | grep 'zpool create'
2023-08-26.18:34:58 zpool create -o ashift=12 Ext /dev/disk/by-id/ata-WDC_WD80EZAZ-11TDBA0_7HKD5KAF
# fdisk -l /dev/disk/by-id/ata-WDC_WD80EZAZ-11TDBA0_7HKD5KAF
The backup GPT table is not on the end of the device.
Disk /dev/sdd: 7.28 TiB, 8001563222016 bytes, 15628053168 sectors
Disk model: WDC WD80EZAZ-11T
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 170F8B4C-BE31-C640-9D5B-112B0B2EAC5A
Device Start End Sectors Size Type
/dev/sdd1 2048 15628034047 15628032000 7.3T Solaris /usr & Apple ZFS
/dev/sdd9 15628034048 15628050431 16384 8M Solaris reserved 1
(Note the first line of the fdisk output, and how there are 15628053168 - 15628050431 = 2737 sectors after the end of sdd9.)
The disks contained all zeroes prior to this as I ran badblocks on them before creating the zpool, so there can't have been any GPT, valid or not, prior.
I don't know if the GPT issues are related to the data corruption whatsoever, but figured I would mention them in case they are.
What are the most likely root cause for the weird file corruption here, and how should I continue to troubleshoot?
My main thought is CPU/RAM (or PSU causing issues with those), but considering how the error happened AFTER the file were written to disk (and how it was okay 5 days after last being written to) I'm not sure how that would explain the symptoms either. I can't come up with ANY explanation that makes sense.
Posted Jan 19, 2024, 11:21 AM UTC