AWS: How to programmatically create a RDS Aurora Cluster in Python/Boto3

Question

My application is hosted on Amazon Web Services, and I'm starting to script the creation of all the infrastructure of my app (VPC, Security Group, Beanstalk ect ...). I did not find the proper way to create a RDS Aurora Cluster, and I failed to reproduce the RDS wizard (helping you to create the db instances and the cluster) in Python with Boto3. Maybe I lack of knowledge in infrastructure, and networks, but I think creating a Aurora cluster must be accessible to me.

So here is my question: Lets says I have a VPC id, a security group id, and some database info (user, password...), what are the minimum API calls I have to do to create a cluster, and make it usable by my application? The procedure must end with a cluster reader/writer endpoint and a reader only endpoint.

Answer 1

Here is how I create an Aurora MySQL instance in Python/BOTO3. You have to implement by yourself some missing functions.

def create_aurora(
    instance_identifier, # used for instance name and cluster name
    db_username,
    db_password,
    db_name,
    db_port,
    vpc_id,
    vpc_sg, # Must be an array
    dbsubnetgroup_name,
    public_access = False,
    AZ = None,
    instance_type = "db.t2.small",
    multi_az = True,
    nb_instance = 1,
    extratags = []
):
    rds = boto3.client('rds')
    # Assume a DB SUBNET Groups exists before creating the cluster. You must have created a DBSUbnetGroup associated to the Subnet of the VPC of your cluster. AWS will find it automatically.

    # 
    # Search if the cluster exists
    try:    
        db_cluster = rds.describe_db_clusters(
            DBClusterIdentifier = instance_identifier
        )['DBClusters']    
        db_cluster = db_cluster[0]
    except botocore.exceptions.ClientError   as e:
        psa.printf("Creating empty cluster\r\n");
        res = rds.create_db_cluster(
            DBClusterIdentifier = instance_identifier,
            Engine="aurora",
            MasterUsername=db_username,
            MasterUserPassword=db_password,
            DBSubnetGroupName=dbsubnetgroup_name,
            VpcSecurityGroupIds=vpc_sg,
            AvailabilityZones=AZ
        )
        db_cluster = res['DBCluster']


    cluster_name = db_cluster['DBClusterIdentifier']
    instance_identifier = db_cluster['DBClusterIdentifier']
    psa.printf("Cluster identifier : %s, status : %s, members : %d\n", instance_identifier , db_cluster['Status'], len(db_cluster['DBClusterMembers']))
    if (db_cluster['Status'] == 'deleting'):
        psa.printf(" Please wait for the cluster to be deleted and try again.\n")
        return None
    psa.printf("   Writer Endpoint : %s\n", db_cluster['Endpoint'])
    psa.printf("   Reader Endpoint : %s\n", db_cluster['ReaderEndpoint'])
    # Now create instances
    # Loop on requested number of instance, and balance them on AZ
    for i in range(1, nb_instance+1):
        if AZ != None:
            the_AZ = AZ[i -1 % len(AZ)]
            dbinstance_id = instance_identifier+"-"+str(i)+"-"+the_AZ
        else:
            the_AZ = None
            dbinstance_id = instance_identifier+"-"+str(i)
        psa.printf("Creating instance %d named '%s' in AZ %s\n", i, dbinstance_id, the_AZ)

        try:
            res = rds.create_db_instance(
                DBInstanceIdentifier=dbinstance_id,
                DBInstanceClass=instance_type,
                Engine='aurora',
                PubliclyAccessible=False,
                AvailabilityZone=the_AZ,
                DBSubnetGroupName=dbsubnetgroup_name,
                DBClusterIdentifier=instance_identifier,
                Tags = psa.tagsKeyValueToAWStags(extratags)
            )['DBInstance']
            psa.printf(" DbiResourceId=%s\n", res['DbiResourceId'])

        except botocore.exceptions.ClientError   as e:
            psa.printf(" Instance seems to exists.\n")
            res = rds.describe_db_instances(DBInstanceIdentifier = dbinstance_id)['DBInstances']
            psa.printf(" Status is %s\n", res[0]['DBInstanceStatus'])
    return db_cluster

Answer 2

Yeah, you are on the right track. Here is the boto3 document for creating a Aurora RDS cluster.

Further, to address the bigger picture problem (ie managing your entire infrastructure as code), you should look at options like Terraform .

Check out their Git Repo Terraform Git Repo So, you can accomplish the same task of creating the Aurora cluster using terraform using this template