I've built a pipeline that just works, do I need to make changes so it uses industry standard tooling?

A few months ago, I didn't have a lot of experience in building ETL pipelines. I was a backend developer and wanted to work on this DE project. The process goes as follows : get data from a paginated API, convert to csv/parquet and store it on azure datalake. This happens once everyday at a fixed time. This would then be used to prepare dashboards by relevant teams.

The setup goes as follows : I created a custom API endpoint using FastAPI and hosted it. When the endpoint is triggered, it creates the file and uploads it to ADLS (Azure data lake storage Gen 2) via it's python SDK. This endpoint is then triggered by make.com(Similar to zapier) at a specific time and then I get an email notification on completion.

Is there a better/more efficient method to do this? So far it's been running smoothly with minimal changes required, that too because the source schema changes sometimes and not because of the setup whatsoever. Fast forward to today, I've learnt more about the tooling within the industry and there seems to be tools like databricks notebooks on azure that also let's me do a similar operation. Is it okay to leave as is?