Migrating Hundreds of ETL Jobs to Airflow – Looking for Experiences & Gotchas

Hi everyone,

We’re planning to migrate our existing ETL jobs to Apache Airflow, starting with the KubernetesPodOperator. The idea is to orchestrate a few hundred (potentially 1-2k) jobs as DAGs in Airflow running on Kubernetes.

A couple of questions for those who have done similar migrations:

• How well does Airflow handle this scale, especially with a high number of DAGs/jobs (1k+)?
• Are there any performance or reliability issues I should be aware of when running this volume of jobs via KubernetesPodOperator?
• What should I pay special attention to when configuring Airflow in this scenario (scheduler, executor, DB settings, etc.)?
• Any war stories or lessons learned (good or bad) you can share?

Any advice, gotchas, or resource recommendations would be super appreciated! Thanks in advance