Scheduling in Data Engineering

Open Source

Hi all!

I'm the developer behind Rocketry, a Pythonic scheduling engine. I originally developed it for other needs than data engineering but as some have said they have started to replace Airflow with it, I thought perhaps some of you might like it as well. I'm also interested in your opinions.

A bit about the framework, it is quite minimal:

from rocketry import Rocketry
from rocketry.conds import daily

app = Rocketry()

@app.task(daily.at("10:00"))
def do_daily():
    ... # This function runs once a day at 10 AM

if __name__ == "__main__":
    app.run()

It has bunch of features: logical scheduling syntax, parallelism/concurrency, dynamic parametization, log to database etc. You can read from the docs: https://rocketry.readthedocs.io/

I don't think Rocketry will achieve the same level of adaptation as Airflow and it is missing many advanced features Airflow has such as built-in UI (working on such), executors for containers etc. but considering how versatile it is and how powerful the scheduling is (basically logical statements), I think it could be interesting for those data engineers who need to customize their setup or need something smaller in scale.

Do you think there is anything missing from the available mature task orchestration products? Or is the space already saturated?

Posted Dec 2, 2022, 7:02 AM UTC

127
kyleekol•

Potentially a very stupid question… but how do you go about deploying the scheduler? Am I right in thinking your main.py script will just need to be continuously running in your backend? Thinking about the costs involved in running this on a cloud server.

This looks great though, a really simple alternative to airflow.

3
Natural-IntelligenceOP•

I don't think that's a stupid question. Perhaps Airflow is more cloud friendly out-of-the-box as it separates the execution from scheduling but I think that's also doable with Rocketry: your task simply calls an API that executes the task itself on more performant cloud instance. That way the (virtual) machine you use for running the scheduler can be quite low-end and cheap. But to be honest, I'm quite beginner with cloud technology so not sure about the details.

I personally run this in two places: on a Windows server and on an Ubuntu virtual machine. Both also handle the task execution so it's not a problem for me at the moment.

3
kyleekol•

Thank you! And thanks u/ratulotron. Yes sorry that’s kind of what I wanted to know, whether the scheduler would/could be deployed on a separate machine. Also very new to this!

2
ratulotronData Engineer•

Any scheduler will have to be run all the time in the background. Otherwise we have to use cron jobs lol. If anything, I think this one would probably cost far less than Airflow or Celery for a basic, minimal setup.

2
ratulotronData Engineer•

Looks interesting! Will test it out in a personal project later, but I am curious about how this is different than Celery+RabbitMQ+Flower setup. From a surface level perspective I can achieve almost the same things with Celery!

1
Natural-IntelligenceOP•

Thanks!

There are some notable differences between Celery and Rocketry. One obvious difference is that Celery doesn't support Windows (though not necessarily a problem for most developers). But I think the main difference is that intent is slightly different. Celery is more of a task queue and Rocketry more of a scheduler. Celery has some options for scheduling but Rocketry was built for it from ground up. On the other hand, Rocketry is not meant to be as high-performance as Celery.

What makes Rocketry different from perhaps all of the options is that its scheduling mechanics is implemented in simple true/false statements. I think this is quite powerful as it enables a lot of fine tuning and customization:

from rocketry.conds import daily, time_of_day, time_of_week

@app.task(daily & (
  (time_of_week.on("Mon") & time_of_day.between("10:00", "11:00")) 
  | (time_of_week.on("Fri") & time_of_day.between("16:00", "17:00")) 
)
def do_things():
    # Runs once on Monday between 10 AM and 11 AM 
    # and once on Friday between 4 PM and 5 PM
    ...

The example is quite non-trivial but the point is that you can combine the scheduling conditions via logical operators (AND, OR, NOT) as you need.

Setting task dependencies is also easy:

from rocketry.conds import after_all_success

@app.task()
def do_a():
    ...

@app.task()
def do_b():
    ...

@app.task(after_all_success(do_a, do_b))
def do_after():
    # Runs once after tasks do_a and do_b both have succeeded
    ...

Or if you wish to have a complete custom condition:

from pathlib import Path
from rocketry.conds import daily

@app.cond()
def file_exists(file):
    return Path(file).is_file()

@app.task(daily & file_exists("myfile.csv"))
def do_things():
    # Runs once a day when myfile.csv exists
    ...

Of course, typically file names of such export files are not static (have timestamp in them). Rocketry also has concept of arguments thus you could create a session level report date and use that in the conditions and also in the tasks themselves.

Some other interesting features of Rocketry:

  • Full async support (integrates well with async web frameworks like FastAPI)
  • Modify the runtime in a task (ie. remove/add tasks or set a task running manually)
  • Flexible logging, currently supports in-memory, CSV, MongoDB and SQL database
  • Various task parametrization/concurrency options (run a task async, threaded or in a separate process)

Sorry, a bit long reply.

2
Main_Tap_1256•

Can the parallel processing task as shown in the docs be run without setting a schedule? Eg as a downstream task?

1