Scheduling in Data Engineering
Hi all!
I'm the developer behind Rocketry, a Pythonic scheduling engine. I originally developed it for other needs than data engineering but as some have said they have started to replace Airflow with it, I thought perhaps some of you might like it as well. I'm also interested in your opinions.
A bit about the framework, it is quite minimal:
from rocketry import Rocketry
from rocketry.conds import daily
app = Rocketry()
@app.task(daily.at("10:00"))
def do_daily():
... # This function runs once a day at 10 AM
if __name__ == "__main__":
app.run()
It has bunch of features: logical scheduling syntax, parallelism/concurrency, dynamic parametization, log to database etc. You can read from the docs: https://rocketry.readthedocs.io/
I don't think Rocketry will achieve the same level of adaptation as Airflow and it is missing many advanced features Airflow has such as built-in UI (working on such), executors for containers etc. but considering how versatile it is and how powerful the scheduling is (basically logical statements), I think it could be interesting for those data engineers who need to customize their setup or need something smaller in scale.
Do you think there is anything missing from the available mature task orchestration products? Or is the space already saturated?
Posted Dec 2, 2022, 7:02 AM UTC
Potentially a very stupid question… but how do you go about deploying the scheduler? Am I right in thinking your main.py script will just need to be continuously running in your backend? Thinking about the costs involved in running this on a cloud server.
This looks great though, a really simple alternative to airflow.
I don't think that's a stupid question. Perhaps Airflow is more cloud friendly out-of-the-box as it separates the execution from scheduling but I think that's also doable with Rocketry: your task simply calls an API that executes the task itself on more performant cloud instance. That way the (virtual) machine you use for running the scheduler can be quite low-end and cheap. But to be honest, I'm quite beginner with cloud technology so not sure about the details.
I personally run this in two places: on a Windows server and on an Ubuntu virtual machine. Both also handle the task execution so it's not a problem for me at the moment.
Thank you! And thanks u/ratulotron. Yes sorry that’s kind of what I wanted to know, whether the scheduler would/could be deployed on a separate machine. Also very new to this!
Any scheduler will have to be run all the time in the background. Otherwise we have to use cron jobs lol. If anything, I think this one would probably cost far less than Airflow or Celery for a basic, minimal setup.