Azure Data Factory Pipeline With 2 copy

Question

Can anyone please tell me how to create a pipeline with two copy activities. Copy activity1 is for InputDataset1 and OutputDataset1 Copy activity2 is for InputDataset2 and OutputDataset2 The pipeline should be scheduled to run both the activities at a time

Answer 1

You simply need to include 2 copy activities in the same pipeline.

Like this:

{
  "name": "Copy2Things",
  "properties": {
    "activities": [
      {
        "type": "Copy",
        "typeProperties": {
          "source": {
            "type": "FileSystemSource",
            "recursive": false
          },
        }
        "inputs": [
          {
            "name": "InputDataset1"
          }
        ],
        "outputs": [
          {
            "name": "OutputDataset1"
          }
        ],
        "policy": {
            //etc...
        },
        "scheduler": {
          "frequency": "Month",
          "interval": 1,
          "style": "EndOfInterval"
        },
        "name": "activity1"
      },
      {
        "type": "Copy",
        "typeProperties": {
          "source": {
            "type": "FileSystemSource",
            "recursive": false
          }
        },
        "inputs": [
          {
            "name": "InputDataset2"
          }
        ],
        "outputs": [
          {
            "name": "OutputDataset2"
          }
        ],
        "policy": {
            //etc...
        },
        "scheduler": {
          "frequency": "Month",
          "interval": 1,
        },
        "name": "activity2"
      }
      //etc...

In terms of them running at the same time ADF will deal with that for you. Or put them in separate pipelines if you want to control each with start/stop/pause options.

Otherwise you can increase the activity concurrency value in the policy block if you want to copy multiple datasets within the scope of the defined time slices at the same time.

Example: InputDataset1, monthly slices, Jan, Feb, Mar, Apr. A concurrency of 2 will copy Jan & Feb, then Mar & Apr in parallel.

Hope this helps.