How do I union a list of lists based on a common string in the lists, Python

Question

I have a list of lists that need to be merged based on strings in the list to fit a structure. In this case, it would be 'date' and 'id' trying to fit the 'fields' structure.

Fields: ['date', 'id', 'impressions', 'clicks']

Before:

[('2015-11-01', 'id123', 'impressions', '8'), ('2015-11-01', 'id123', 
'clicks', '4'), ('2015-11-01', 'id456', 'impressions', '14'), 
('2015-11-01', 'id456', 'clicks', '9')]

After:

[('2015-11-01', 'id123', '8', '4'), ('2015-11-01', 'id456', '14', '9')]

Answer 1

>>> L  = [('2015-11-01', 'id123', 'impressions', '8'), ('2015-11-01', 'id123', 
... 'clicks', '4'), ('2015-11-01', 'id456', 'impressions', '14'), 
... ('2015-11-01', 'id456', 'clicks', '9')]
>>> from collections import defaultdict
>>> D = defaultdict(list)
>>> for a, b, c, d in L:
...     D[a, b].append(d)
... 
>>> [k + tuple(D[k]) for k in D]
[('2015-11-01', 'id456', '14', '9'), ('2015-11-01', 'id123', '8', '4')]

In the case that impressions and clicks are not in a consistent order

>>> L = [('2015-11-01', 'id123', 'impressions', '8'), ('2015-11-01', 'id123', 'clicks', '4'), ('2015-11-01', 'id456', 'clicks', '9'), ('2015-11-01', 'id456', 'impressions', '14')]
>>> from collections import defaultdict
>>> D = defaultdict(lambda: [None, None])
>>> for a, b, c, d in L:
...     D[a, b][c == 'clicks'] = d
... 
>>> [k + tuple(D[k]) for k in D]
[('2015-11-01', 'id456', '14', '9'), ('2015-11-01', 'id123', '8', '4')]

Answer 2

itertools.groupby can work well here, particularly if the real data matches the sample data (already sorted so date/id pairs are all adjacent):

import itertools
from operator import itemgetter

outlist = []
for (date, ID), grp in itertools.groupby(inlist, key=itemgetter(0, 1)):
    grp = list(grp)  # Iterating twice, so convert to sequence
    impressioncnt = sum(int(cnt) for _, _, typ, cnt in grp if typ == 'impressions')
    clickcnt = sum(int(cnt) for _, _, typ, cnt in grp if typ == 'clicks')
    outlist.append((date, ID, str(impressioncnt), str(clickcnt)))

If the data isn't already sorted by date and ID , you'd need to sort the inlist first, inlist.sort(key=itemgetter(0, 1)) . That could be expensive if the list is huge, in which case you might consider using a collections.defaultdict instead:

import collections

dateID_cnts = collections.defaultdict({'impressions': 0, 'clicks': 0}.copy)
for date, ID, typ, cnt in inlist:
    dateID_cnts[date, ID][typ] += int(cnt)

# Convert from defaultdict to desired list of tuples
outlist = [(date, ID, str(v['impressions']), str(v['counts'])) for (date, ID), v in dateID_cnts.items()]

Answer 3

Another way:

data=[('2015-11-01', 'id123', 'impressions', '8'), 
      ('2015-11-01', 'id123','clicks', '4'), 
      ('2015-11-01', 'id456', 'impressions', '14'), 
      ('2015-11-01', 'id456', 'clicks', '9')]

ddict={}
for t in data:
    key=(t[0], t[1])
    ddict.setdefault(key, []).append(t[2:])

LoT=[]    
for d, id in ddict:
    impressions, clicks=max(ddict[(d, id)])[1], min(ddict[(d, id)])[1]
    LoT.append(tuple([d, id, impressions, clicks]))

>>> LoT
[('2015-11-01', 'id123', '8', '4'), ('2015-11-01', 'id456', '14', '9')]

If you can assume that impressions and clicks are already in order, you can eliminate max and min and replace that line with:

impressions, clicks=ddict[(d, id)][0][1], ddict[(d, id)][1][1]