Fudges knowledge in OoTP

Sorry if this has been discussed before, but I haven't seen it if it has. At Harry's trial in the beginning of OoTP, Fudge sarcastically mentions a reversal of time as one of Harry's ridiculous stories. At first I thought this referred to the third book. However, Fudge was never informed about the time travel bit. That was just to save Sirius and Buckbeak, and he obviously was not informed about them. So what is he referring to? Or is this just an oversight?