aws数据管道日期时间变量

Question

I am using AWS Data Pipeline to save a text file to my S3 bucket from RDS. I would like the file name to to have the date and the hour in the file name like:

myfile-YYYYMMDD-HH.txt
myfile-20140813-12.txt

I have specified my S3DataNode FilePath as:

s3://mybucketname/out/myfile-#{format(myDateTime,'YYYY-MM-dd-HH')}.txt

When I try to save my pipeline I get the following error:

ERROR: Unable to resolve myDateTime for object:DataNodeId_xOQxz

According to the AWS Data Pipeline documentation for date and time functions this is the proper syntax for using the format function.

When I save pipeline using a "hard-coded" the date and time I don't get this error and my file is in my S3 bucket and folder as expected.

My thinking is that I need to define "myDateTime" somewhere or use a NOW()

Can somebody tell me how to set "myDateTime" to the current time (eg NOW) or give a workaround so I can format the current time to be used in my FilePath ?

Answer 1

I am not aware of an exact equivalent of NOW() in Data Pipeline. I tried using makeDate with no arguments (just for fun) to see if that worked.. it did not.

The closest are runtime variables scheduledStartTime, actualStartTime, reportProgressTime.

http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-object-s3datanode.html

The following for eg. should work. s3://mybucketname/out/myfile-#{format(@scheduledStartTime,'YYYY-MM-dd-HH')}.txt

Answer 2

Just for fun, here is some more info on Parameters .

At the end of your Pipeline Json (click List Pipelines , select into one, click Edit Pipeline , then click Export ), you need to add a Parameters and/or Values object.

I use a myStartDate for backfill processes which you can manipulate once it is passed in for ad hoc runs. You can give this a static default, but can't set it to a dynamic value so it is limited for regular schedule tasks. For realtime/scheduled dates, you need to use the @scheduledStartTime , etc, as suggested. Here is a sample of setting up some Parameters and or Values . Both show up in Parameters in the UI. These values can be used through out your pipeline activities (shell, hive, etc) with the #{myVariableToUse} notation.

"parameters": [
{
  "helpText": "Put help text here",
  "watermark": "This shows if no default or value set",
  "description": "Label/Desc",
  "id": "myVariableToUse",
  "type": "string"
}
]

And for Values:

"values": {
  "myS3OutLocation": "s3://some-bucket/path",
  "myThreshold": "30000",
}

You cannot add these directly in the UI (yet) but once they are there you can change and save the values.