将Python列表分组在一起以获取公共元素

Question

I'm querying Google Analytics data for sessions and users for each different country. I want to save this data in my db for each single day so I can access it later on.

My query gives me a really big json back and I'm trying to find the optima solution to maximise speed.

First of all I managed to get back the data ordered by sessions, which means that I can now save only the first 10 countries in my db without saving for each day a new row for each country.

I think this is the minimum amount of data I need in order to have valuable info. So now I structured my bd to accept data like this:

20170101 | US | 112 (sessions) | 111 (users)
20170101 | CA | 111 (sessions) | 221 (users)
... (for 8 more rows)
20170102 | US | 11 (sessions) | 22 (users)
... (and so on, so 10 rows per day)

Now my big json that I get back looks something like this (I've removed a lot of metrics in the middle):

m = {
'reports': [{
    'data': {
        'rowCount': 2003,
        'maximums': [{
            'values': ['1219', '1109']
        }],
        'minimums': [{
            'values': ['1', '1']
        }],
        'totals': [{
            'values': ['33505', '30382']
        }],
        'rows': [{
            'dimensions': ['20170404', 'US'],
            'metrics': [{
                'values': ['1219', '1091']
            }]
        }, {
            'dimensions': ['20170406', 'US'],
            'metrics': [{
                'values': ['1203', '1109']
            }]
        }, {
            'dimensions': ['20170405', 'US'],
            'metrics': [{
                'values': ['1185', '1073']
            }]
        }, {
            'dimensions': ['20170408', 'PL'],
            'metrics': [{
                'values': ['2', '1']
            }]
        }, {
            'dimensions': ['20170408', 'SG'],
            'metrics': [{
                'values': ['2', '2']
            }]
        }, {
            'dimensions': ['20170408', 'TT'],
            'metrics': [{
                'values': ['2', '2']
            }]
        }]
    },
    'nextPageToken': '1000',
    'columnHeader': {
        'dimensions': ['ga:date', 'ga:countryIsoCode'],
        'metricHeader': {
            'metricHeaderEntries': [{
                'name': 'ga:sessions',
                'type': 'INTEGER'
            }, {
                'name': 'ga:users',
                'type': 'INTEGER'
            }]
        }
    }
}]
}

I'm trying to figure out how I can extract the top 10 countries with most sessions for each day and save this info in my db, so far I came up with:

x = m['reports'][0]['data']['rows']

l =[]
for data in x:
    date = data['dimensions'][0]
    country = data['dimensions'][1]
    sessions = data['metrics'][0]['values'][0]
    users = data['metrics'][0]['values'][1]
    n = [date, [country,sessions, users]]
    l.append(n)

This generates me a list with inside values in the format [date[country, sessions, users]]

so something like this:

[['20170404', ['US', '1219', '1091']],
 ['20170406', ['US', '1203', '1109']],
 ['20170405', ['US', '1185', '1073']],
 ['20170408', ['PL', '2', '1']],
 ['20170408', ['SG', '2', '2']],
 ['20170408', ['TT', '2', '2']]]

Now I was thinking to nest an other for loop which checks the date and if it's the same it will add the values z[1] to the same list, so for every date I would have a list with the values for each single country. However I'm not sure how to group these dictionary together according the the first value z[0] plus this would do all the countries and not only the top 10 ones.

Is there an easier way to accomplish this given the big json above? If how do I group lists together according to the first value and how I then sort by sessions?

Thanks!

Answer 1

When there are no duplicate countries per day. You could use defaultdicts , to mange the different levels of grouping (magically):

import pprint
from collections import defaultdict

def recursive_defaultdict():
    return defaultdict(recursive_defaultdict)

l = recursive_defaultdict()

x = m['reports'][0]['data']['rows']

for data in x:
    date = data['dimensions'][0]
    country = data['dimensions'][1]
    sessions = data['metrics'][0]['values'][0]
    users = data['metrics'][0]['values'][1]

    l[date][country] = {'sessions': sessions, 'users': users}

pprint.pprint(l)

This returns a dict, that allows you to easily iterate over:

defaultdict(<function recursive_defaultdict at 0x7f3ecfb45e18>,
            {'20170404': defaultdict(<function recursive_defaultdict at 0x7f3ecfb45e18>,
                                     {'US': {'sessions': '1219',
                                             'users': '1091'}}),
             '20170405': defaultdict(<function recursive_defaultdict at 0x7f3ecfb45e18>,
                                     {'US': {'sessions': '1185',
                                             'users': '1073'}}),
             '20170406': defaultdict(<function recursive_defaultdict at 0x7f3ecfb45e18>,
                                     {'US': {'sessions': '1203',
                                             'users': '1109'}}),
             '20170408': defaultdict(<function recursive_defaultdict at 0x7f3ecfb45e18>,
                                     {'PL': {'sessions': '2', 'users': '1'},
                                      'SG': {'sessions': '2', 'users': '2'},
                                      'TT': {'sessions': '2', 'users': '2'}})})

To receive a specific combination of date/country:

print (l['20170404']['US'])
>>> {'sessions': '1219', 'users': '1091'}

Iterate through result:

for date, values in l.items():
    for country, value in values.items():
        print (date, country, value)