使用 aws cli 创建 Kinesis Analytics 应用程序

Question

I want to create a kinesis analytics application using aws cli. I use this command to create the application

aws kinesisanalytics create-application --application-name smartfactorytest1 --application-code "CREATE OR REPLACE STREAM DESTINATION_SQL_STREAM ( "device_serial" VARCHAR(16), "uploadRate"  INTEGER, "downloadRate" INTEGER);
CREATE OR REPLACE PUMP "STREAM_PUMP" 
    AS INSERT INTO DESTINATION_SQL_STREAM
SELECT STREAM  "device_serial",  "uploadRate", "downloadRate"
    FROM SOURCE_SQL_STREAM_001
-- LIKE compares a string to a string pattern (_ matches all char, % matches substring)
-- SIMILAR TO compares string to a regex, may use ESCAPE
    WHERE "uploadRate" >20000" --inputs NamePrefix="SOURCE_SQL_STREAM",KinesisStreamsInput={ResourceARN="sourcearn",RoleARN="rolearn"}

But I get this error

invalid type for parameter Inputs[0].KinesisStreamsInput, value: ResourceARN=string, type: <class 'str'>, valid types: <class 'dict'>

Can anyone tell me what am I doing wrong? Any help would be appreciated.

Answer 1

I believe the issue is either that you need to take the quotes out in the KinesisStreamsInput section, or you need to add quotes and escape them. The documentation is unclear on which is the correct option.

1. According to the AWS Kinesis Analytics CLI Reference, https://docs.aws.amazon.com/cli/latest/reference/kinesisanalytics/create-application.html , the syntax for --inputs with KinesisStreamsInput should look like the example provided for KinesisStreamsOutput :

    Name=string,KinesisStreamsOutput={ResourceARN=string,RoleARN=string},...

   This would mean removing the quotes around your sourcearn and rolearn . However, the documentation isn't clear that this refers to the CLI syntax in all cases.

2. If that doesn't work, according to this AWS CLI usage guide page, https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-parameters-quoting-strings.html , it specifies adding quotes and escaping the relevant ones, depending on your OS...

   "Linux or macOS

   Use single quotation marks (' ') to enclose the JSON data structure, as in the following example. You don't have to do anything special with the embedded double quotation marks embedded in the JSON string.

    aws ec2 run-instances --image-id ami-12345678 --block-device-mappings '[{"DeviceName":"/dev/sdb","Ebs":{"VolumeSize":20,"DeleteOnTermination":false,"VolumeType":"standard"}}]'

   PowerShell

   PowerShell requires single quotation marks (' ') to enclose the JSON data structure. Also, because double quotation marks have a special meaning to PowerShell, you must use a backslash () to escape each double quotation mark (") within the JSON structure, as in the following example.

    PS C:\\> aws ec2 run-instances --image-id ami-12345678 --block-device-mappings '[{\\"DeviceName\\":\\"/dev/sdb\\",\\"Ebs\\":{\\"VolumeSize\\":20,\\"DeleteOnTermination\\":false,\\"VolumeType\\":\\"standard\\"}}]'

   Windows Command Prompt

   The Windows command prompt requires double quotation marks (" ") to enclose the JSON data structure. Also, to prevent the command processor from misinterpreting the double quotation marks embedded in the JSON, you must also escape (precede with a backslash [ \\ ] character) each double quotation mark (") within the JSON data structure itself, as in the following example.

    C:\\> aws ec2 run-instances --image-id ami-12345678 --block-device-mappings "[{\\"DeviceName\\":\\"/dev/sdb\\",\\"Ebs\\":{\\"VolumeSize\\":20,\\"DeleteOnTermination\\":false,\\"VolumeType\\":\\"standard\\"}}]"

   Only the outermost double quotation marks are not escaped."

3. This link also references needing to escape quotes on Windows, and is using the kinesisanalytics command: https://github.com/aws/aws-cli/issues/3103

   "Rishi74744 commented on Feb 6, 2018

   I got it to work as -

    aws kinesisanalytics add-application-reference-data-source --endpoint https://kinesisanalytics.us-east-1.amazonaws.com --region us-east-1 --application-name alerts --reference-data-source "{\\"TableName\\":\\"DeviceData\\",\\"S3ReferenceDataSource\\":{\\"BucketARN\\":\\"arn: aws: s3: : : bucket-name\\",\\"FileKey\\":\\"device.csv\\",\\"ReferenceRoleARN\\":\\"arn: aws: iam: : account-id: role/role-name\\"},\\"ReferenceSchema\\":{\\"RecordFormat\\":{\\"RecordFormatType\\":\\"CSV\\",\\"MappingParameters\\":{\\"CSVMappingParameters\\":{\\"RecordRowDelimiter\\":\\"\\n\\",\\"RecordColumnDelimiter\\":\\", \\"}}},\\"RecordEncoding\\":\\"UTF-8\\",\\"RecordColumns\\":[{\\"Name\\":\\"key1\\",\\"SqlType\\":\\"VARCHAR(64)\\"},{\\"Name\\":\\"key2\\",\\"SqlType\\":\\"VARCHAR(64)\\"}]}}" --current-application-version-id 2

   But this should be mentioned in the documentation."

4. One note: it may be preferable to use JSON files as inputs and use this syntax instead: --cli-input-json file://input.json . This is referenced in the AWS Kinesis CLI Command Reference (first link, under 1.) and also mentioned in the GitHub link above. It's also the method used by the majority of the AWS Kinesis documentation. For example, JSON files used for different purposes in Kinesis Analytics: https://docs.aws.amazon.com/kinesisanalytics/latest/dev/how-it-works-input.html

Please let me know what works, and I will work with my AWS rep to improve the documentation.