You know, those things that no matter how far you dig with “Five Whys” or RCAs that it comes down to something that just shouldn’t be technically possible. Legitimately speculating that the only explanation is random bit flips…
My short story is we had a production Oracle database randomly get its time set back a few hours. We tracked the system logs to its check-in with our internal NTP server saying it was a few hours ahead so it adjusted accordingly. The thing is the NTP server itself was 100% stable with nothing odd in its log and none of the other 300 servers using it were affected. My RCA was since NTP is over UDP there must have been some corruption of the NTP packet on the wire caus