How to run multiple files in one container image and use the output as input in another container in Argo workflow?

Question

I am trying to run 2 python files in one container, save the output as parameters:

- name: training
    serviceAccountName: argo-service
    outputs:
      parameters:
        - name: output-param-1
          valueFrom:
            path: ./tmp/output.txt
        - name: output-param-2
          valueFrom:
            path: ./tmp/output2.txt
    container:
      image: (image name)
      command: [python]
      args: ["/data/argo/model_build.py","/data/argo/model_build_2.py"]

Use that output as input in another container:

- name: evaluate
    serviceAccountName: argo-service
    inputs:
        parameters:
          - name: value-1
          - name: value-2
    container:
      image: (image name)
      command: ["python", "/data/argo/evaluate.py"]
      args: 
          - '{{inputs.parameters.value-1}}'
          - '{{inputs.parameters.value-2}}'

and have defined the chain as:

 - name: dag-chain
    dag:
      tasks:
      - name: src
        template: clone-repo
      - name: prep
        template: data-prep
        dependencies: [src]
      - name: train
        template: training
        dependencies: [prep]
      - name: eval
        template: evaluate
        dependencies: [train]
        args:
          parameters:
            - name: value-1
              value: '{{dag-chain.tasks.train.outputs.parameters.output-param-1}}'
            - name: value-2
              value: '{{dag-chain.tasks.train.outputs.parameters.output-param-2}}'

But with these steps I'm getting the error:

" Internal Server Error: templates.dag-chain.tasks.eval templates.evaluate inputs.parameters.value-1 was not supplied: "

Please help me identify the mistakes I'm making.

I have tried the steps mentioned above but it's not working.

Answer 1

I don't have Argo accessible just now to test, but a couple of things to try:

• args in DAG tasks should be arguments (see field reference for a DAG task here ).
• Try removing dag-chain from the parameters (see example here ).

 - name: dag-chain
    dag:
      tasks:
      - name: src
        template: clone-repo
      - name: prep
        template: data-prep
        dependencies: [src]
      - name: train
        template: training
        dependencies: [prep]
      - name: eval
        template: evaluate
        dependencies: [train]
        arguments:
          parameters:
            - name: value-1
              value: '{{tasks.train.outputs.parameters.output-param-1}}'
            - name: value-2
              value: '{{tasks.train.outputs.parameters.output-param-2}}'

If that doesn't work I'll try some more steps with Argo.