Python create list of dictionaries from csv on S3

Question

I am trying to take a CSV and create a list of dictionaries in python with the CSV coming from S3. Code is as follows:

import os
import boto3
import csv
import json
from io import StringIO
import logging
import time

s3 = boto3.resource('s3')
s3Client = boto3.client('s3','us-east-1')

bucket = 'some-bucket'
key = 'some-key'

obj = s3Client.get_object(Bucket = bucket, Key = key)
lines = obj['Body'].read().decode('utf-8').splitlines(True)

newl = []

for line in csv.reader(lines, quotechar='"', delimiter=',',quoting=csv.QUOTE_ALL,skipinitialspace=True, escapechar="\\"):
    newl.append(line)

fieldnames = newl[0]
newl1 = newl[1:]

reader = csv.DictReader(newl1,fieldnames)
out = json.dumps([row for row in reader])
jlist1 = json.loads(out)

but this gives me the error:

iterator should return strings, not list (did you open the file in text mode?)

if I alter the for loop to this:

for line in csv.reader(lines, quotechar='"', delimiter=',',quoting=csv.QUOTE_ALL,skipinitialspace=True, escapechar="\\"):
    newl.append(','.join(line))

then it works, however there are some fields that have commas in them so this completely screws up the schema and shifts the data. For example:

|address1   |address2  |state|
------------------------------
|123 Main st|APT 3, Fl1|TX   |

becomes:

|address1   |address2  |state|null|
-----------------------------------
|123 Main st|APT 3     |Fl1  |TX  |

Where am I going wrong?

Answer 1

The problem is that you are building a list of lists here :

 newl.append(line)

and as the error says : iterator should return strings, not list

so try to cast line as a string:

newl.append(str(line))

Hope this helps :)

Answer 2

I ended up changing the code to this:

obj = s3Client.get_object(Bucket = bucket, Key = key)
lines1 = obj['Body'].read().decode('utf-8').split('\n')
fieldnames = lines1[0].replace('"','').split(',')
testls = [row for row in csv.DictReader(lines1[1:], fieldnames)]
out = json.dumps([row for row in testls])
jlist1 = json.loads(out)

And got the desired result