How can I run more complex job dependencies via Rundeck

Question

I'm currently testing Rundeck for some PoCs I'm working on. I currently only have it running on a single machine via docker, so I don't have access to multiple nodes.

I want to simulate the following job dependencies:

JobA is needed for JobB and JobC. But JobB and JobC must run in parallel.

JobC must run if the current local time is 9:00am. Doesn't matter if Job A is finished/started yet or not.

All this should be able to easily scale and expand up to several hundreds of Jobs.

Can some of you guys possibly help me? I have tried several configurations including the Plug-in Job State. I somehow can't get this to work. The best I can do is run all the jobs either sequential or parallel.

Answer 1

The best (and easiest) way to achieve that is to use the ruleset strategy (only available on PagerDuty Process Automation On Prem, formerly "Rundeck Enterprise") calling your jobs using the job reference step .

So, on the community version, the "workaround" is to call the jobs from a Parent Job via Rundeck API , which basically is about scripting the custom workflow behavior, eg :

JobA

- defaultTab: nodes
  description: ''
  executionEnabled: true
  id: b156b3ed-fde6-4fdc-af81-1ee919edc4ff
  loglevel: INFO
  name: JobA
  nodeFilterEditable: false
  plugins:
    ExecutionLifecycle: null
  scheduleEnabled: true
  sequence:
    commands:
    - exec: whoami
    keepgoing: false
    strategy: node-first
  uuid: b156b3ed-fde6-4fdc-af81-1ee919edc4ff

JobB

- defaultTab: nodes
  description: ''
  executionEnabled: true
  id: 4c46c795-20c9-47a8-aa2e-e72f183b150f
  loglevel: INFO
  name: JobB
  nodeFilterEditable: false
  plugins:
    ExecutionLifecycle: null
  scheduleEnabled: true
  sequence:
    commands:
    - exec: whoami
    keepgoing: false
    strategy: node-first
  uuid: 4c46c795-20c9-47a8-aa2e-e72f183b150f

JobC (runs every day at 9.00 AM, no matter the JobA execution).

- defaultTab: nodes
  description: ''
  executionEnabled: true
  id: ce45c4d1-1350-407b-8001-2d90daf49eaa
  loglevel: INFO
  name: JobC
  nodeFilterEditable: false
  plugins:
    ExecutionLifecycle: null
  schedule:
    month: '*'
    time:
      hour: '09'
      minute: '00'
      seconds: '0'
    weekday:
      day: '*'
    year: '*'
  scheduleEnabled: true
  sequence:
    commands:
    - exec: whoami
    keepgoing: false
    strategy: node-first
  uuid: ce45c4d1-1350-407b-8001-2d90daf49eaa

ParentJob (all the behavior logic is inside a bash script, needs jq tool to work)

- defaultTab: nodes
  description: ''
  executionEnabled: true
  id: 6e67020c-7293-4674-8d56-5db811ae5745
  loglevel: INFO
  name: ParentJob
  nodeFilterEditable: false
  plugins:
    ExecutionLifecycle: null
  scheduleEnabled: true
  sequence:
    commands:
    - description: A friendly starting message
      exec: echo "starting..."
    - description: Execute JobA and Get the Status
      fileExtension: .sh
      interpreterArgsQuoted: false
      plugins:
        LogFilter:
        - config:
            invalidKeyPattern: \s|\$|\{|\}|\\
            logData: 'true'
            regex: ^(job_a_exec_id)\s*=\s*(.+)$
          type: key-value-data
      script: "# execute JobA\nexec_id=$(curl -s -X POST \\\n  \"http://localhost:4440/api/41/job/b156b3ed-fde6-4fdc-af81-1ee919edc4ff/run\"\
        \ \\\n  --header \"Accept: application/json\" \\\n  --header \"X-Rundeck-Auth-Token:\
        \ 45hNfblPiF6C1l9CfJeG37l08oAgs0Gd\" | jq .id)\n  \necho \"job_a_exec_id=$exec_id\""
      scriptInterpreter: /bin/bash
    - description: Time to execute the Job
      exec: sleep 2
    - description: 'Get the execution Status '
      fileExtension: .sh
      interpreterArgsQuoted: false
      plugins:
        LogFilter:
        - config:
            invalidKeyPattern: \s|\$|\{|\}|\\
            logData: 'false'
            regex: ^(job_a_exec_status)\s*=\s*(.+)$
          type: key-value-data
      script: "exec_status=$(curl -s -X GET \\\n  'http://localhost:4440/api/41/execution/@data.job_a_exec_id@'\
        \ \\\n  --header 'Accept: application/json' \\\n  --header 'X-Rundeck-Auth-Token:\
        \ 45hNfblPiF6C1l9CfJeG37l08oAgs0Gd' | jq -r .status)\n  \necho \"job_a_exec_status=$exec_status\""
      scriptInterpreter: /bin/bash
    - description: 'Run JobB and JobC pararelly, depending on JobA Execution Status'
      fileExtension: .sh
      interpreterArgsQuoted: false
      script: "# Now execute JobB and JobC depending on the JobA final status (dependency)\n\
        \nif [ @data.job_a_exec_status@ = \"succeeded\" ]; then\n  echo \"Job A execition\
        \ is OK, running JobB and JobC pararelly...\"\n \n  # JobB \n  curl -X POST\
        \ \\\n  \"http://localhost:4440/api/41/job/4c46c795-20c9-47a8-aa2e-e72f183b150f/run\"\
        \ \\\n  --header \"Accept: application/json\" \\\n  --header \"X-Rundeck-Auth-Token:\
        \ 45hNfblPiF6C1l9CfJeG37l08oAgs0Gd\" &\n  \n  # JobC\n  curl -X POST \\\n\
        \  \"http://localhost:4440/api/41/job/ce45c4d1-1350-407b-8001-2d90daf49eaa/run\"\
        \ \\\n  --header \"Accept: application/json\" \\\n  --header \"X-Rundeck-Auth-Token:\
        \ 45hNfblPiF6C1l9CfJeG37l08oAgs0Gd\" &\nelse\n  echo \"Job A has failed, dependencey\
        \ failed\"\nfi"
      scriptInterpreter: /bin/bash
    - description: A friendly ending message
      exec: echo "done."
    keepgoing: false
    strategy: node-first
  uuid: 6e67020c-7293-4674-8d56-5db811ae5745

Parent Job's JobB/JobC launching script:

# Now execute JobB and JobC depending on the JobA final status (dependency)

if [ @data.job_a_exec_status@ = "succeeded" ]; then
  echo "Job A execition is OK, running JobB and JobC pararelly..."
 
  # JobB 
  curl -X POST \
  "http://localhost:4440/api/41/job/4c46c795-20c9-47a8-aa2e-e72f183b150f/run" \
  --header "Accept: application/json" \
  --header "X-Rundeck-Auth-Token: 45hNfblPiF6C1l9CfJeG37l08oAgs0Gd" &
  
  # JobC
  curl -X POST \
  "http://localhost:4440/api/41/job/ce45c4d1-1350-407b-8001-2d90daf49eaa/run" \
  --header "Accept: application/json" \
  --header "X-Rundeck-Auth-Token: 45hNfblPiF6C1l9CfJeG37l08oAgs0Gd" &
else
  echo "Job A has failed, dependencey failed"
fi