(ETL) Transformation in Fabric

So I’ve just started as a Data Engineer in a company that landed on Microsoft Fabric as they’re data warehouse. I am the only data engineer at the company, and the prior job has been executed by former employee.

I’m writing this post in order to get a better understanding of compute resources and to make the transformation faster and easier (more efficient).

There are bronze, silver and gold lakehouses, populated by using spark notebooks, wich are orchestrated by pipelines.

For the ingest, there are some sources ingested trough Dataflow Gen2, the rest are notebooks. From bronze to silver, there are approximately 20 notebooks that transforms data type, and unpacks json files into 20 tables in silver. From silver to gold there are approximately 30 notebooks that joins, aggregate and filters the tables into different dim and fact tables used creating 3 semantic models (for BI). The biggest fact table has approximately 30 million rows.

I find it really hard to keep up with all the notebooks, as multiple of the notebooks are between 500-1000 lines of code. Earlier I used to work with snowflake and databricks, where the transformation was done in DBT. Now, I can’t see the lineage, and when I need to attach a new source it is a lot of coding needed in order to join it to the existing solution. (My Claude subscription has gone trough the roof).

What is the best way to efficiently transform data from bronze to gold layer? For future models, and sources I am free to handle it the way I want. And at my current pyspark level, I’m finding it time consuming to copy the existing work.

Also, in the creation of semantic models we run into capacity issues when multiple notebooks run at the same time (F32). Would it be easier to do transformation with T-sql or dataflows?

DBT has a really nice lineage feature, when trying to access the lineage in fabrics, I can only see the notebooks and they’re lakehouses. In order to schooling other non-data employees at the firm, it would be great to show them the lineage of table connections!

While waiting on answers Il be busy learning pyspark on YouTube! Thanks!