my offsite backup silently failed for weeks and every manual test said it was fine

pull based setup, a mac at home fetches encrypted dumps from my remote boxes on a launchd schedule. finally looked at the log and 11 of the last 25 runs had failed, the newest one pulled 0 files. the boxes were backing themselves up fine the whole time, only the copy off them was dead.

root cause was dumb in hindsight. the script borrowed my logged in ssh agent, and that agent empties on every reboot. so after any restart the scheduled run had no keys and gave up, while every test i ran from a terminal passed because my shell had the keys loaded. fix was making the script load its own keys instead of leaning on my session.

so now i check the schedulers own exit codes and count files on the receiving side instead of trusting a run from my shell.

what do people actually monitor for backup jobs? i had "job ran" checks and it still sat half broken for weeks