Tested our disaster recovery plan for the first time in 2 years - here's what we found and it wasn't pretty

Finally stopped procrastinating and ran a full DR test last month. Thought it'd be a quick formality. It was not.

The highlights:

- Backups were running fine. Restores were silently failing for months. Green checkmarks the whole time.

- Our recovery runbook referenced 3 servers we decommissioned and a vendor we haven't used since 2022

- Nobody actually knew their role when it came down to it. Everyone waited for someone else to move first

- We promised leadership a 4 hour RTO. Actual test took 9 hours. In a calm controlled environment.

Nothing real was lost, no actual incident - that's the point of testing. But we had been completely comfortable for two years thinking we were covered.

If you haven't actually tested a restore recently, not just checked that the backup job is green, do it this week.

Anyone else find surprises when they finally ran a real test?