How to get boto3 to display _all_ RDS instances?

Question

Trying to get all RDS instances with boto3 - does not return all RDS instances.

When I look at my RDS instances in Oregon (us-west-2), I see the following:

However, if I run the below Python3 script, I only get one result:

$ python3 ./stackoverflow.py 

RDS instances in Oregon
------------------------------
aurora-5-7-yasmin.cazdggrmkpt1.us-west-2.rds.amazonaws.com qa test db.t2.small aurora-5-7-yasmin
$ 

Can you suggest a way to get boto3 to display all RDS instances?

---

$ cat ./stackoverflow.py 
import collections
import boto3
import datetime
import pygsheets

REGIONS = ('us-west-2',)
REGIONS_H = ('Oregon',)

currentDT = str(datetime.datetime.now())


def create_spreadsheet(outh_file, spreadsheet_name = "AWS usage"):
    client = pygsheets.authorize(outh_file=outh_file, outh_nonlocal=True)
    client.list_ssheets(parent_id=None)
    spread_sheet = client.create(spreadsheet_name)
    return spread_sheet


def rds_worksheet_creation(spread_sheet):
    for i in range(len(REGIONS)):
        region = REGIONS[i]
        region_h = REGIONS_H[i]
        print()
        print("{} instances in {}".format("RDS", region_h))
        print("------------------------------")

        client = boto3.client('rds', region_name=region)
        db_instances = client.describe_db_instances()
        for i in range(len(db_instances)):
            j = i - 1
            try:
                DBName = db_instances['DBInstances'][j]['DBName']
                MasterUsername = db_instances['DBInstances'][0]['MasterUsername']
                DBInstanceClass = db_instances['DBInstances'][0]['DBInstanceClass']
                DBInstanceIdentifier = db_instances['DBInstances'][0]['DBInstanceIdentifier']
                Endpoint = db_instances['DBInstances'][0]['Endpoint']
                Address = db_instances['DBInstances'][0]['Endpoint']['Address']
                print("{} {} {} {} {}".format(Address, MasterUsername, DBName, DBInstanceClass,
                DBInstanceIdentifier))
            except KeyError:
                continue


if __name__ == "__main__":
    spread_sheet = create_spreadsheet(spreadsheet_name = "AWS usage", outh_file = '../client_secret.json')
    spread_sheet.link(syncToCloud=False)
    rds_worksheet_creation(spread_sheet)

$ cat ../client_secret.json 
{"installed":{"client_id":"362799999999-uml0m2XX4v999999mr2s03XX9g8l9odi.apps.googleusercontent.com","project_id":"amiable-shuttle-198516","auth_uri":"https://accounts.google.com/o/oauth2/auth","token_uri":"https://accounts.google.com/o/oauth2/token","auth_provider_x509_cert_url":"https://www.googleapis.com/oauth2/v1/certs","client_secret":"XXXXxQH434Qg-xxxx99_n0vW","redirect_uris":["urn:ietf:wg:oauth:2.0:oob","http://localhost"]}}
$ 

Edit 1:

Following Michael's comment, I changed the script to the following, but even though one more related line appeared, most of the RDS instances are still not returned:

$ python3 ./stackoverflow.py 

RDS instances in Oregon
------------------------------
aurora-5-7-yasmin.cazdggrmkpt1.us-west-2.rds.amazonaws.com qa +++ DBName gave KeyError +++ db.t2.small aurora-5-7-yasmin
aurora-5-7-yasmin.cazdggrmkpt1.us-west-2.rds.amazonaws.com qa test db.t2.small aurora-5-7-yasmin
$ 

$ cat ./stackoverflow.py 
import collections
import boto3
import datetime
import pygsheets

REGIONS = ('us-west-2',)
REGIONS_H = ('Oregon',)

currentDT = str(datetime.datetime.now())


def create_spreadsheet(outh_file, spreadsheet_name = "AWS usage"):
    client = pygsheets.authorize(outh_file=outh_file, outh_nonlocal=True)
    client.list_ssheets(parent_id=None)
    spread_sheet = client.create(spreadsheet_name)
    return spread_sheet


def rds_worksheet_creation(spread_sheet):
    for i in range(len(REGIONS)):
        region = REGIONS[i]
        region_h = REGIONS_H[i]
        print()
        print("{} instances in {}".format("RDS", region_h))
        print("------------------------------")

        client = boto3.client('rds', region_name=region)
        db_instances = client.describe_db_instances()
        for i in range(len(db_instances)):
            j = i - 1
            try:
                DBName = db_instances['DBInstances'][j]['DBName']
            except KeyError:
                DBName = "+++ DBName gave KeyError +++"
            MasterUsername = db_instances['DBInstances'][0]['MasterUsername']
            DBInstanceClass = db_instances['DBInstances'][0]['DBInstanceClass']
            DBInstanceIdentifier = db_instances['DBInstances'][0]['DBInstanceIdentifier']
            Endpoint = db_instances['DBInstances'][0]['Endpoint']
            Address = db_instances['DBInstances'][0]['Endpoint']['Address']
            print("{} {} {} {} {}".format(Address, MasterUsername, DBName, DBInstanceClass,
            DBInstanceIdentifier))


if __name__ == "__main__":
    spread_sheet = create_spreadsheet(spreadsheet_name = "AWS usage", outh_file = '../client_secret.json')
    spread_sheet.link(syncToCloud=False)
    rds_worksheet_creation(spread_sheet)

Answer 1

You have an error in your original code but if you want this code to scale to a large number of instances (it is unlikely you'll need this) then you'll want to use something like the following:

import boto3
available_regions = boto3.Session().get_available_regions('rds')

for region in available_regions:
    rds = boto3.client('rds', region_name=region)
    paginator = rds.get_paginator('describe_db_instances').paginate()
    for page in paginator:
        for dbinstance in page['DBInstances']:
            print("{DBInstanceClass}".format(**dbinstance))

You can get rid of the paginator and just use the first loop if you know each region will have fewer than 100s of instances:

for region in available_regions:
    rds = boto3.client('rds', region_name=region)
    for dbinstance in rds.describe_db_instances():
        print("{DBInstanceClass}".format(**dbinstance))

Additionally you can provide a simple

dbinstance.get('DBName', 'No Name Set')

instead of excepting around the KeyError.

Answer 2

Your for loop range is getting the value of 2 since db_instances is dict type.

Instead of

for i in range(len(db_instances)):

It should be

for i in range(len(db_instances['DBInstances'])):

Which gives list type and correct length to iterate the loop.

Answer 3

This Code will list all RDS instances present in the account

Try this 100 % working code

#!/usr/bin/env python
import boto3
client = boto3.client('rds')
response = client.describe_db_instances()
for i in response['DBInstances']:
   db_name = i['DBName']
   db_instance_name = i['DBInstanceIdentifier']
   db_type = i['DBInstanceClass']
   db_storage = i['AllocatedStorage']
   db_engine = i['Engine']
   print db_instance_name,db_type,db_storage,db_engine

Answer 4

FYI, the more Pythonic way to do loops in this case would be:

for instance in db_instances['DBInstances']:
    MasterUsername = instance['MasterUsername']
    DBInstanceClass = instance['DBInstanceClass']
    etc.

This avoids the need for i -type iterators.