具有多个输入的AWS数据管道活动

Question

As part of an Amazon AWS data pipeline, I have a hive activity using two unstaged S3 data nodes as input. What I want is to be able to set two script variables on the activity, each pointing to an input data node, but I can't get the syntax right. With the single input, I could write the following and it would work just fine:

INPUT_FOO=#{input.directoryPath}

When I add the second input, I run into a problem of how to reference them since they are now an array of inputs, as you can see in the pipeline definition below. Essentially, I want to achieve the following, but can't figure out the correct syntax:

INPUT_FOO=#{input[1].directoryPath}
INPUT_BAR=#{input[2].directoryPath}

Here's the activity portion of the pipeline definition:

{
  "id": "ActivityId_7u1sR",
  "input": [
    {
      "ref": "DataNodeId_iYnxf"
    },
    {
      "ref": "DataNodeId_162Ka"
    }
  ],
  "schedule": {
    "ref": "DefaultSchedule"
  },
  "scriptUri": "#{myS3ScriptLocation}calculate-results.q",
  "name": "Perform Calculations",
  "runsOn": {
    "ref": "EmrClusterId_jHeiV"
  },
  "scriptVariable": [
    "INPUT_SOURCE1=#{input[1].directoryPath}",
    "OUTPUT=#{output.directoryPath}Results/",
    "INPUT_SOURCE2=#{input[2].directoryPath}"
  ],
  "output": {
    "ref": "DataNodeId_2jY6v"
  },
  "type": "HiveActivity",
  "stage": "false"
}

I plan to keep the tables unstaged and take care of table creation in the hive script so that it's easier to run each Hive activity in isolation as well as in the pipeline itself.

Here's the error I see when using array syntax:

Unable to resolve input[1].directoryPath for object ActivityId_7u1sR'

Answer 1

从目前的情况来看，不支持这种情况，但是将来添加了功能请求以支持它。