优化在python中合并两个字典列表

Question

i have two lists, each one contains 10-15k dictionaries. the first list named campaigns and it contains dictionaries with the following structure:

{u'campaign_id': u'400037', u'ctr': u'1.1210', u'roi': 0, u'end_date': None, u'revenue': 0.0, u'website_id': 1, u'enabled': u'active', u'budget': u'100.00', u'default_bid': u'0.05', u'cost': 30.63, u'start_date': u'2018-02-13'}

and the second list - yesterday_data dict's structure is:

{u'cost': 0.0, u'campaign_id': u'400037', u'revenue': 0.0}

goal is to match campaign id's, and add two key's to the relevant dict in campaigns_data where yesterday_date["revenue"] and yesterday_date["cost"] will be a new keys in campaigns_data dict - yesterday_cost and yesterday_revenue

i managed to a achieve this logic with the following code:

campaigns = model.get_campaigns_data(self.mysql_db)
yesterday_data = model.get_yesterday_data(self.mysql_db, yesterday)
try:
    for campaign in campaigns:
        missing = filter(lambda c: c["campaign_id"] == str(campaign["campaign_id"]), yesterday_data)
        if not missing:
            pass
        else:
            campaign["yesterday_spend"] = missing[0]["cost"]
            campaign["yesterday_revenue"] = missing[0]["revenue"]

but with this numbers of dictionaries inside each list it's extremely slow and ,i want to believe, far fro being the optimized way to achieve that. any idea how can i improve my code to get the same result?

Answer 1

Group by id and reduce:

grouper = {}
for d in campaigns_data:
    grouper[d["campaign_id"]] = d

# assuming the keys match up:

for d in yesterday_data:
    grouper[["campaign_id"]].update(yesterday_spend=d['cost'],
                                    yesterday_revenue=d['revenue'])

Answer 2

Try:

>>> d1 = {u'campaign_id': u'400037', u'ctr': u'1.1210', u'roi': 0, u'end_date': None, u'revenue': 0.0, u'website_id': 1, u'enabled': u'active', u'budget': u'100.00', u'default_bid': u'0.05', u'cost': 30.63, u'start_date': u'2018-02-13'}
>>> d2 = {u'cost': 0.0, u'campaign_id': u'400037', u'revenue': 0.0}

>>> if d1["campaign_id"] == d2["campaign_id"]:
      d1.update(d2)

Output

{u'roi': 0, u'ctr': u'1.1210', u'end_date': None, u'revenue': 0.0, u'website_id': 1, u'enabled': u'active', u'campaign_id': u'400037', u'budget': u'100.00', u'default_bid': u'0.05', u'cost': 0.0, u'start_date': u'2018-02-13'}

Answer 3

Use pandas to do this. Convert both these dictionaries to pandas dataframe and then merge them.

Note: I am doing this for one row. If you have 100 rows replace the index=[0] to index = range(100) in the lines where I am constructing df_1 and df_2

campaigns = {u'campaign_id': u'400037', u'ctr': u'1.1210', u'roi': 0, u'end_date': None, u'revenue': 0.0, u'website_id': 1, u'enabled': u'active', u'budget': u'100.00', u'default_bid': u'0.05', u'cost': 30.63, u'start_date': u'2018-02-13'}

yesterday_data = {u'cost': 0.0, u'campaign_id': u'400037', u'revenue': 0.0}

import pandas as pd
df1 = pd.DataFrame(campaigns, index=[0])
df2 = pd.DataFrame(yesterday_data, index=[0])

df_new = df1.merge(df2, on=['campaign_id'])

to get back the dictionary form, do:

df_new.to_dict(orient='records')

Note 2 : The keys from yesterday_data for example, for example cost and revenue will end with _y as suffix and for keys from campaigns will have keys with suffix _x . This is for the keys which are common between the two dataframes.