I have an application which runs with a http server in the background. It will receives jobs with one or multiple images as data (also videos in the future) and has to process them step by step (modify images, create archive, ftp upload and a few more steps).
Multiple jobs can arrive at nearly the same time. (or another can arrive while one is still under processing). Each processing step should only exist once in the system as they can be heavy on recources (either network I/O or CPU e.g.).
I'd like to create a structure for the application (which can be flexible on the steps for each job), which schedules the steps from the jobs in multiple queues and has multiple processes which wor