Job orchestration

Question

I am currently redoing all our database orchestration jobs (ETL, backups, daily tasks, report compilation, etc.)

To do that, I would need a task/job orchestrator where I can define tasks dependency, time based tasks, async tasks, etc.

I was looking at celery and Flow Based Programming technologies but I am not sure these are good for my use case.

I am looking more at a framework that would support all these things out of the box. Any suggestions?

Answer 1

Probably to late, but I wanted to mention Job runner for possibly other people arriving at this question.

From their GitHub repository README.rst:

Job-Runner is a crontab like tool, with a nice web-frontend for administration and (live) monitoring the current status.

Features:

• Schedule recurring jobs
• Chaining of jobs
• Load-balance workers by putting them in a pool
• Schedule jobs to run on all workers within a pool
• Live dashboard (with option to kill runs and ad-hoc scheduling)
• Multiple projects and per-project permission management