How do you handle standardized ingestion pipelines on Databricks?

Curious how people are solving this in their teams.

Every time I set up a new ingestion pipeline on Databricks, I end up writing a lot of the same boilerplate — handling incrementals, schema evolution, basic data quality checks, etc. And across a team, everyone ends up doing it slightly differently.

I've been looking at tools like dlt (data load tool) and dlt-meta, but I'm wondering what the broader community is doing:

- Do you write everything from scratch?

- Do you use an internal framework your team built?

- Do you rely on a specific tool (dlt, ADF, Fivetran, Airbyte...)?

- Have you found a clean way to make this config-driven (YAML or similar) so that spinning up a new source is just filling in a file?

Especially interested in hearing from people who've built something internal — what did you solve that off-the-shelf tools didn't cover?

Thanks