是否可以通过提供json文件中的所有配置在EMR中创建集群

Question

I want to automate the cluster creation task in EMR. I have a json file which contains configurations which need to be applied on new cluster and I want to write a shell script which automates this task for me.

Is it possible to create cluster in EMR by giving all the configurations from json file?

For example I have this file

{
"Cluster": { 
    "Ec2InstanceAttributes": {
        "EmrManagedMasterSecurityGroup": "sg-00b10b71", 
        "RequestedEc2AvailabilityZones": [], 
        "AdditionalSlaveSecurityGroups": [], 
        "AdditionalMasterSecurityGroups": [], 
        "RequestedEc2SubnetIds": [
            "subnet-02291b3e"
        ], 
        "Ec2SubnetId": "subnet-02291b3e", 
        "IamInstanceProfile": "EMR_EC2_DefaultRole", 
        "Ec2KeyName": "perf_key_pair", 
        "Ec2AvailabilityZone": "us-east-1e", 
        "EmrManagedSlaveSecurityGroup": "sg-f2b30983"
    }, 
    "Name": "NitinJ-Perf", 
    "ServiceRole": "EMR_DefaultRole", 
    "Tags": [
        {
            "Value": "Perf-Nitink", 
            "Key": "Qubole"
        }
    ], 
    "Applications": [
        {
            "Version": "3.7.2", 
            "Name": "Ganglia"
        }, 
        {
            "Version": "2.7.3", 
            "Name": "Hadoop"
        }, 
        {
            "Version": "2.1.1", 
            "Name": "Hive"
        }, 
        {
            "Version": "0.16.0", 
            "Name": "Pig"
        }, 
        {
            "Version": "0.8.4", 
            "Name": "Tez"
        }
    ], 
    "MasterPublicDnsName": "ec2-34-229-254-217.compute-1.amazonaws.com", 
    "ScaleDownBehavior": "TERMINATE_AT_INSTANCE_HOUR", 
    "InstanceGroups": [
        {
            "RequestedInstanceCount": 4, 
            "Status": {
                "Timeline": {
                    "ReadyDateTime": 1499150835.979, 
                    "CreationDateTime": 1499150533.99
                }, 
                "State": "RUNNING", 
                "StateChangeReason": {
                    "Message": ""
                }
            }, 
            "Name": "Core Instance Group", 
            "InstanceGroupType": "CORE", 
            "EbsBlockDevices": [], 
            "ShrinkPolicy": {}, 
            "Id": "ig-34P3CVF8ZL5CW", 
            "Configurations": [], 
            "InstanceType": "r3.4xlarge", 
            "Market": "ON_DEMAND", 
            "RunningInstanceCount": 4
        }, 
        {
            "RequestedInstanceCount": 1, 
            "Status": {
                "Timeline": {
                    "ReadyDateTime": 1499150804.591, 
                    "CreationDateTime": 1499150533.99
                }, 
                "State": "RUNNING", 
                "StateChangeReason": {
                    "Message": ""
                }
            }, 
            "Name": "Master Instance Group", 
            "InstanceGroupType": "MASTER", 
            "EbsBlockDevices": [], 
            "ShrinkPolicy": {}, 
            "Id": "ig-3V7EHQ36187PY", 
            "Configurations": [], 
            "InstanceType": "r3.4xlarge", 
            "Market": "ON_DEMAND", 
            "RunningInstanceCount": 1
        }
    ], 
    "Configurations": [
        {
            "Properties": {
                "hive.vectorized.execution.enabled": "true"
            }, 
            "Classification": "hive-site"
        }
    ]
}
}

Can I create a cluster on EMR by using some command like

aws emr create-cluster --cli-input-json file://'pwd'/emr_cluster_up.json

Answer 1

There is no such option through the AWS CLI as per the AWS CLI documentation. But if you want to automate the EMR cluster creation using a JSON file. You can use Cloud formation and automate the cluster creation.

Getting Started with AWS CloudFormation