How to format a DynamoDB scan in Python Boto3 to be human readable to html?

Question

I have a python boto3 app that is running in flask. The main goal is to format the list retrieval to be human readable. As in: I want to have the list return as:

Employee Name        Employee ID
 Finn                  1001
 Jake                  1002
 Bubblegum             1003
 Marceline             1004
 BMO                   1005

and so on.

Currently, when I get the response for the full table retrieval, it looks like this:

[{'EmployeeID': Decimal('1007'), 'Employee Name': 'Lumpy Space Princess'}, {'EmployeeID': Decimal('1021'), 'Employee Name': 'Glob'}, {'EmployeeID': Decimal('1029'), 'Employee Name': 'Snail'}, {'EmployeeID': Decimal('1038'), 'Employee Name': 'Banana Guard #216'}, {'EmployeeID': Decimal('1010'), 'Employee Name': 'Martin Mertens'}, {'EmployeeID': Decimal('1015'), 'Employee Name': 'Patience St. Pim'}, {'EmployeeID': Decimal('1035'), 'Employee Name': 'Cinnamon Bun'}, {'EmployeeID': Decimal('1002'), 'Employee Name': 'Jake'}, {'EmployeeID': Decimal('1018'), 'Employee Name': 'GOLB'}, {'EmployeeID': Decimal('1008'), 'Employee Name': 'Flame Princess'}, {'EmployeeID': Decimal('1032'), 'Employee Name': 'Jermaine'}, {'EmployeeID': Decimal('1014'), 'Employee Name': 'Earl of Lemongrab'}, {'EmployeeID': Decimal('1022'), 'Employee Name': 'Grod'}, {'EmployeeID': Decimal('1009'), 'Employee Name': 'Lady Rainicorn'}, {'EmployeeID': Decimal('1034'), 'Employee Name': 'Margaret'}, {'EmployeeID': Decimal('1023'), 'Employee Name': 'King of Mars'}, {'EmployeeID': Decimal('1001'), 'Employee Name': 'Finn'}, {'EmployeeID': Decimal('1025'), 'Employee Name': 'Tiny Manticore'}, {'EmployeeID': Decimal('1005'), 'Employee Name': 'Marceline The Vampire Queen'}, {'EmployeeID': Decimal('1005'), 'Employee Name': 'Marceline the Vampire Queen'}, {'EmployeeID': Decimal('1017'), 'Employee Name': 'Uncle Gumbald'}, {'EmployeeID': Decimal('1036'), 'Employee Name': 'Starchy'}, {'EmployeeID': Decimal('1026'), 'Employee Name': 'Magic Man'}, {'EmployeeID': Decimal('1037'), 'Employee Name': 'Banana Guard #1'}, {'EmployeeID': Decimal('1011'), 'Employee Name': 'Betty Grof'}, {'EmployeeID': Decimal('1012'), 'Employee Name': 'King of Ooo'}, {'EmployeeID': Decimal('1030'), 'Employee Name': 'Huntress Wizard'}, {'EmployeeID': Decimal('1006'), 'Employee Name': 'BMO'}, {'EmployeeID': Decimal('1031'), 'Employee Name': 'Forest Spirit'}, {'EmployeeID': Decimal('1016'), 'Employee Name': 'Fern'}, {'EmployeeID': Decimal('1033'), 'Employee Name': 'Joshua '}, {'EmployeeID': Decimal('1003'), 'Employee Name': 'Bonnibel Bubblegum'}, {'EmployeeID': Decimal('1024'), 'Employee Name': 'Death'}, {'EmployeeID': Decimal('1020'), 'Employee Name': 'Gob'}, {'EmployeeID': Decimal('1027'), 'Employee Name': 'Prismo'}, {'EmployeeID': Decimal('1019'), 'Employee Name': 'Grob'}, {'EmployeeID': Decimal('1028'), 'Employee Name': 'Gunter'}, {'EmployeeID': Decimal('1013'), 'Employee Name': 'Hunson Abadeer'}]

It is only one element. It is both un-ordered, and has all the meta data. All my googling, and I can't find out how to extract the actual data fields to be formatted into a list for HTML human-readable compatibility.

My retrieval looks like this:

resource = boto3.resource('dynamodb',
aws_access_key_id="AAAAAAAAAAAAAAAAAAAA",
aws_secret_access_key="BBBBBBBBBBBBBBBBBBBBBB",
region_name='us-east-1')
    table = resource.Table('external-data')

    response = table.scan()
    data = response['Items']

while 'LastEvaluatedKey' in response:
response = table.scan(ExclusiveStartKey=response['LastEvaluatedKey'])
        data.append(response['Items'])

    ansExit = ''
    return str(data)

I have tried creating a list and appending each response as an element, but this response is only one element. I think I need to break this up in an array such that I can print it to HTML as arr[i,j] where arr[1,1] = "1001", arr[1,2] = "Finn", arr[2,1] = "1002", arr[2,2] = "Jake", and so on. Is this the right direction to go? I'm not even sure is this possible, or if this just simply isn't how DynamoDB data is meant to be read.

I just want the name and employee number to be isolated so I can print them without the metadata.

Thank you.

Answer 1

You could use the pandas package:

import pandas as pd
from decimal import Decimal

data = [{'EmployeeID': Decimal('1007'), 'Employee Name': 'Lumpy Space Princess'}, {'EmployeeID': Decimal('1021'), 'Employee Name': 'Glob'}, {'EmployeeID': Decimal('1029'), 'Employee Name': 'Snail'}, {'EmployeeID': Decimal('1038'), 'Employee Name': 'Banana Guard #216'}, {'EmployeeID': Decimal('1010'), 'Employee Name': 'Martin Mertens'}, {'EmployeeID': Decimal('1015'), 'Employee Name': 'Patience St. Pim'}, {'EmployeeID': Decimal('1035'), 'Employee Name': 'Cinnamon Bun'}, {'EmployeeID': Decimal('1002'), 'Employee Name': 'Jake'}, {'EmployeeID': Decimal('1018'), 'Employee Name': 'GOLB'}, {'EmployeeID': Decimal('1008'), 'Employee Name': 'Flame Princess'}, {'EmployeeID': Decimal('1032'), 'Employee Name': 'Jermaine'}, {'EmployeeID': Decimal('1014'), 'Employee Name': 'Earl of Lemongrab'}, {'EmployeeID': Decimal('1022'), 'Employee Name': 'Grod'}, {'EmployeeID': Decimal('1009'), 'Employee Name': 'Lady Rainicorn'}, {'EmployeeID': Decimal('1034'), 'Employee Name': 'Margaret'}, {'EmployeeID': Decimal('1023'), 'Employee Name': 'King of Mars'}, {'EmployeeID': Decimal('1001'), 'Employee Name': 'Finn'}, {'EmployeeID': Decimal('1025'), 'Employee Name': 'Tiny Manticore'}, {'EmployeeID': Decimal('1005'), 'Employee Name': 'Marceline The Vampire Queen'}, {'EmployeeID': Decimal('1005'), 'Employee Name': 'Marceline the Vampire Queen'}, {'EmployeeID': Decimal('1017'), 'Employee Name': 'Uncle Gumbald'}, {'EmployeeID': Decimal('1036'), 'Employee Name': 'Starchy'}, {'EmployeeID': Decimal('1026'), 'Employee Name': 'Magic Man'}, {'EmployeeID': Decimal('1037'), 'Employee Name': 'Banana Guard #1'}, {'EmployeeID': Decimal('1011'), 'Employee Name': 'Betty Grof'}, {'EmployeeID': Decimal('1012'), 'Employee Name': 'King of Ooo'}, {'EmployeeID': Decimal('1030'), 'Employee Name': 'Huntress Wizard'}, {'EmployeeID': Decimal('1006'), 'Employee Name': 'BMO'}, {'EmployeeID': Decimal('1031'), 'Employee Name': 'Forest Spirit'}, {'EmployeeID': Decimal('1016'), 'Employee Name': 'Fern'}, {'EmployeeID': Decimal('1033'), 'Employee Name': 'Joshua '}, {'EmployeeID': Decimal('1003'), 'Employee Name': 'Bonnibel Bubblegum'}, {'EmployeeID': Decimal('1024'), 'Employee Name': 'Death'}, {'EmployeeID': Decimal('1020'), 'Employee Name': 'Gob'}, {'EmployeeID': Decimal('1027'), 'Employee Name': 'Prismo'}, {'EmployeeID': Decimal('1019'), 'Employee Name': 'Grob'}, {'EmployeeID': Decimal('1028'), 'Employee Name': 'Gunter'}, {'EmployeeID': Decimal('1013'), 'Employee Name': 'Hunson Abadeer'}]

df = pd.DataFrame(data)
df.head(5)

Your output would look like:

    EmployeeID  Employee Name
0   1007        Lumpy Space Princess
1   1021        Glob
2   1029        Snail
3   1038        Banana Guard #216
4   1010        Martin Mertens

You could even create HTML if you need that:

df.head().to_html()

Output:

'<table border="1" class="dataframe">\n  <thead>\n    <tr style="text-align: right;">\n      <th></th>\n      <th>EmployeeID</th>\n      <th>Employee Name</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1007</td>\n      <td>Lumpy Space Princess</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>1021</td>\n      <td>Glob</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>1029</td>\n      <td>Snail</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>1038</td>\n      <td>Banana Guard #216</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>1010</td>\n      <td>Martin Mertens</td>\n    </tr>\n  </tbody>\n</table>'