Newbie to DE needs help with the approach to the architecture of a project

Help

So I was hired as a data analyst a few months ago and I have a background in software development. A few months ago I was moved to a smallish project with the objective of streamlining some administrative tasks that were all calculated "manually" with Excel. By the time, all I had worked with were very basic, low code tools from the Microsoft enviroment: PBI for dashboards, Power Automate, Power Apps for data entry, Sharepoint lists, etc, so that's what I used to set it up.

The cost for the client is basically nonexistent right now, apart from a couple of PBI licenses. The closest I've done to ETL work has been with power query, if you can even call it that.

Now I'm at a point where it feels like that's not gonna cut it anymore. I'm going to be working with larger volumes of data, with more complex relationships between tables and transformations that need to be done earlier in the process. I could technically keep going with what I have but I want to actually build something durable and move towards actual data engineering, but I don't know where to start with a solution that's cost efficient and well structured. For example, I wanted to move the data from Sharepoint lists to a proper database but then we'd have to pay for multiple premium licenses to be able to connect to them in powerapps. Where do I even start?

I know the very basics of data engineering and I've done a couple of tutorial projects with Snowflake and Databricks as my team seems to want to focus on cloud based solutions. So I'm not starting from absolute scratch, but I feel pretty lost as I'm sure you can tell. I'd appreciate any kind of advice or input as to where to head from here, as I'm on my own right now.

Posted Apr 1, 2025, 11:00 AM UTC

05
Loading comments…