Where do you draw the line between “Pythonic modules” and a plugin runtime?

I’m refactoring a Python control plane that runs long-lived, failure-prone workloads (AI/ML pipelines, agents, execution environments).

This project started in a very normal Python way: modules, imports, helper functions, direct composition. It was fast to build and easy to change early on.

Then the system got bigger, and the problems became very practical:

• a pipeline crashes in the middle and leaves part of the system initialized
• cleanup is inconsistent (or happens in the wrong order)
• shared state leaks between runs
• dependencies are spread across imports/helpers and become hard to reason about
• no clean way to say “this component can access X, but not Y”

I didn’t move to plugins because I wanted a framework. I moved because failure cleanup kept biting me, and the same class of issues kept coming back.

So I moved the core to a plugin runtime with explicit lifecycle and dependency boundaries.

What changed:

• components implement a plugin contract (initialize() / shutdown())
• lifecycle is managed by the runtime (not by whatever caller remembered to do)
• dependencies are resolved explicitly (graph-based)
• components get scoped capabilities instead of broad/raw access

It helped a lot with reliability and isolation.

But now even small tasks need extra structure (manifests/descriptors, lifecycle hooks, capability declarations). In Python, that definitely feels heavier than just writing a module and importing it.

Question

For people building orchestrators / control planes / platform-like systems in Python:

Where did you draw the line between:

• lightweight Python modules + conventions
• and a managed runtime / container / plugin architecture?

If you stayed with a lighter approach, what patterns gave you reliable lifecycle/cleanup/isolation without building a full plugin runtime?

(Attached 3 small snippets to show the general shape of the plugin contract + manifest-based loading, not the full system.)

English isn’t my first language, so sorry if some wording is awkward.