Can't import local Python modules in multi-node GPU cluster on Azure Databricks
Hello,
I have the following cluster: Multi-node GPU (NC4as_T4_v3) with runtime 16.1 ML + Unity Catalog enabled.
I cloned my repo in Repos:
my-repo/
├── notebook.ipynb
└── utils/
├── __init__.py
└── my_module.py
In notebook.ipynb, I run:
from utils.my_module import some_function
- which works fine on CPU and serverless clusters. But on the GPU cluster, I get
ModuleNotFoundError. sys.path looks fine (repo root is there)s - os.listdir('.') and dbutils.fs.ls('.') return empty
Same exact code works on CPU. Is this a GPU-specific limitation(& if so, why) or security feature? Or a bug? Can’t find anything about this in the Databricks docs.
Thanks,
Posted Jul 22, 2025, 8:02 PM UTC
61