Every Python developer relies on __pycache__ without thinking about it. The first import pandas of the day is slow, every one after it is fast, because CPython compiled the .py files to bytecode once and kept the result.

The invalidation rule behind that is the whole subject of this post. A timestamp-based .pyc records the mtime and size of the source file it was compiled from. On import, CPython compares that recorded mtime against the source file's current mtime. If they differ, the cached bytecode is thrown away and the module is recompiled. That is correct, and it is why editing a file takes effect immediately.

It also means the cache is worth nothing in any environment where the source files get re-created between runs. If something in your pipeline unpacks packages fresh, restores a dependency cache without preserving mtimes, or otherwise rewrites the files, then every recorded mtime is wrong the moment it arrives, every cached .pyc is discarded, and you pay full compilation every single time. Nothing tells you this is happening.

What it costs, measured on CPython 3.14:

import pandas              2.3s
import matplotlib.pyplot   1.9s
import numpy               0.5s

Almost none of that is the package doing anything. It is CPython compiling about a thousand .py files into bytecode that is byte-for-byte identical to the bytecode it produced a moment earlier in another process.

PEP 552 is the fix, and it has been there since 3.7

A .pyc can be hash-based rather than timestamp-based: it records a hash of the source instead of the source's mtime. mtime stops entering the validity check entirely, so re-creating a file with a new timestamp no longer invalidates anything. This is close to the original motivation for PEP 552, since an installer cannot promise anything about the mtimes of the files it has just written either.

PEP 552 defines two flavours, and the choice matters more than it looks:

• checked: re-reads and re-hashes the source on every import to confirm the hash still matches.
• unchecked: taken on trust, the source is not read at all.

Checked sounds like the responsible option, and it defeats the purpose: re-hashing the source on every import is most of the I/O you were trying to avoid. Unchecked is correct when you can bound what it trusts. A released package's files do not change while its version stays the same, so key the cache on package name plus version and that claim stays true.

The conversion is also cheaper than people expect. Turning a timestamp-based .pyc into a hash-based one is header surgery, not compilation. The marshalled code object, which is the entire expensive part, is byte-identical; only the 16-byte header changes. Rewriting the whole tree for numpy and pandas took 115ms.

The sys.pycache_prefix trap

sys.pycache_prefix (3.8+) redirects bytecode into a tree of its own instead of __pycache__ directories sitting next to your source. Useful any time you do not want those directories showing up inside someone's source tree.

It has a failure mode worth knowing because it is completely silent. CPython builds the directory tree under the prefix by walking up from the .pyc's intended directory until it finds a directory that already exists. If that walk runs off the top without finding one, it starts creating directories relative to the current working directory instead, reports nothing, and no bytecode ever lands where you are looking for it.

The fix is one line: make sure the prefix root exists before the first import.

One rule worth keeping

Cache installed packages, never the user's own modules. A released package's files do not change. Your own files change constantly, and a file you just edited must never be at risk of running as a stale copy. Leave your own code on CPython's ordinary mtime checking, where an edit always wins.

A cache that is occasionally wrong about your own source is worse than no cache, by a margin that no amount of saved seconds closes.