Python - Generator object into a list of lists

Question

I have a generator object 'results', which when looping through returns a list of dictionaries. I'm trying to convert this into a list of lists, so I can easily loop through and reference each value to be INSERTed into a database. I believe I'm having trouble because this is a generator object, how can I do this?

such as:

def parse(results):
    for r in results:
        print(r)

results:

[{'id': '7229957054', 'repost_of': None, 'name': '1996 Acura Integra', 'url': 'https://monterey.craigslist.org/cto/d/salinas-1996-acura-integra/7229957054.html', 'datetime': '2020-11-12 14:37', 'last_updated': '2020-11-12 14:37', 'price': '$1,000', 'where': 'Salinas', 'has_image': True, 'geotag': None, 'deleted': False}, {'id': '7229839309', 'repost_of': None, 'name': '1990 Acura Integra GS', 'url': 'https://monterey.craigslist.org/cto/d/salinas-1990-acura-integra-gs/7229839309.html', 'datetime': '2020-11-12 11:31', 'last_updated': '2020-11-12 11:31', 'price': '$2,800', 'where': 'Salinas, Ca', 'has_image': True, 'geotag': None, 'deleted': False}]

my code:

def initialParse(results):
    rList = []
    for r in results:
        r_id = str(r['id'])
        r_name = str(r['name'])
        r_url = str(r['url'])
        r_datetime = str(r['datetime'])
        r_updated = str(r['last_updated'])
        r_price = str(r['price'])
        r_where = str(r['where'])
        iList = list(r_id + r_name + r_url + r_datetime + r_updated + r_price + r_where)
        rList.append(iList)
    print(rList)

returns:

[['7', '2', '2', '9', '9', '5', '7', '0', '5', '4', '1', '9', '9', '6', ' ', 'A', 'c', 'u', 'r', 'a', ' ', 'I', 'n', 't', 'e', 'g', 'r', 'a', 'h', 't', 't', 'p', 's', ':', '/', '/', 'm', 'o', 'n', 't', 'e', 'r', 'e', 'y', '.', 'c', 'r', 'a', 'i', 'g', 's', 'l', 'i', 's', 't', '.', 'o', 'r', 'g', '/', 'c', 't', 'o', '/', 'd', '/', 's', 'a', 'l', 'i', 'n', 'a', 's', '-', '1', '9', '9', '6', '-', 'a', 'c', 'u', 'r', 'a', '-', 'i', 'n', 't', 'e', 'g', 'r', 'a', '/', '7', '2', '2', '9', '9', '5', '7', '0', '5', '4', '.', 'h', 't', 'm', 'l', '2', '0', '2', '0', '-', '1', '1', '-', '1', '2', ' ', '1', '4', ':', '3', '7', '2', '0', '2', '0', '-', '1', '1', '-', '1'...]

Moving the rList.append() out one block gives a list in a list containing ALL entries... I need each result r in it's own list inside a list... like this:

[['id', 'name', 'url', 'datetime', 'lastupdated', 'price', 'where'], ['id', 'name', 'url', 'datetime', 'lastupdated', 'price', 'where'], ... ]

what am I doing incorrectly here?

Answer 1

Look like you are trying to get this:

[list(r.keys()) for r in results]

That being said, there might be a better way to do what you're doing; could you explain more about how you plan to INSERT the items? Or do you want the values in the lists:

[list(r.values()) for r in results]

But really, it looks like you are dealing with structured data here, which means you should be adding it into a pandas dataframe and filtering rows or selecting columns as needed. All you need to do is:

import pandas as pd
df = pd.DataFrame(results)

You can read up on the pandas documentation to see how to operate on the data and get the table you want. It would also make it much easier to ultimately load into your SQL database.

Answer 2

Replace

iList = list(r_id + r_name ...)

with

iList = [r_id, r_name, ...]

r_id + r_name... creates a big string by concatenating all the arguments, and the list() function creates a list of all characters in the string. You don't want that.