Lost files from a mirror zfs pool?
Hi folks,
I have a strange situation here, it's not a catastrophe so I'm hoping to learn from this and would like to know what I did wrong.
I've set up a zfs mirror pool using two 8TB WD Red drives I've pulled from my NAS, and I'm just using this pool to store backups. I also have a Proxmox Backup Server running as a VM, storing the data on an iSCSI LUN on the NAS, so the loss of the local backup storage is not a big deal.
A few days ago I've received a notification that the pool has degraded due to too many errors on one of the drives. The guides I've found suggested checking the drive's smart data (which looked good) and then clearing the error (zpool clear) and then data scrubbing (zpool scrub).
Before the scrubbing the status of the pool showed that it was degraded, but without data loss, and the other drive in the pool is healthy. After the scrubbing I have a whole bunch of files with permanent errors. I thought this was impossible on a mirror pool, that with one drive failure the pool would still continue to work without loss of data:
>root@proxmox:~# zpool status hdd -v
pool: hdd
state: DEGRADED
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
scan: scrub repaired 623G in 18:59:14 with 17690 errors on Thu Apr 2 11:28:40 2026
config:
NAME STATE READ WRITE CKSUM
hdd DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
ata-WDC_WD80EFAX-68LHPN0_7SJNPXYW DEGRADED 0 0 29.6M too many errors
ata-WDC_WD80EFAX-68KNBN0_VAJ77PAL ONLINE 0 0 26.7M
errors: Permanent errors have been detected in the following files:
/hdd/backup/dump/vzdump-lxc-101-2026_03_29-01_00_06.tar.zst
/hdd/backup/dump/vzdump-lxc-101-2026_02_22-01_00_07.tar.zst
/hdd/backup/dump/vzdump-lxc-105-2026_03_29-01_11_57.tar.zst
/hdd/backup/dump/vzdump-lxc-100-2026_03_29-01_00_01.tar.zst
/hdd/backup/dump/vzdump-lxc-102-2026_03_29-01_00_11.tar.zst
/hdd/backup/dump/vzdump-lxc-103-2026_03_29-01_08_00.tar.zst
hdd:<0x98a>
/hdd/backup/dump/vzdump-qemu-104-2026_03_29-01_11_16.vma.zst
/hdd/backup/dump/vzdump-lxc-105-2026_02_22-01_12_20.tar.zst
/hdd/backup/dump/vzdump-lxc-102-2026_02_22-01_00_11.tar.zst
/hdd/backup/dump/vzdump-lxc-100-2026_02_22-01_00_01.tar.zst
/hdd/backup/dump/vzdump-lxc-103-2026_02_22-01_08_22.tar.zst
/hdd/backup/dump/vzdump-qemu-104-2026_02_22-01_11_48.vma.zst
Both drives show up as healthy under disks, sda is the one that's degraded:
The degraded drive shows 0 uncorrectable errors, 0 udma crc errors:
Apart from obviously replacing the drive, I would like to know what did I do wrong to lose data in a mirrored pool and how to avoid it in the future?
Posted Apr 5, 2026, 10:22 AM UTC