Pandas Dataframe：基于字典列表更新多行的最快方法

Question

What is the fastest way of replacing values in multiple rows of a pandas DataFrame based on a list of dictionaries (messages). Eventually, I want to process real time streaming data (from a websocket connection) at a peak rate of ~1000 messages per second. Below I post a simplified artificial example to illustrate the task. At the moment the performance is not fast enough.

import random
names = ["Jim", "Bryan", "Roy", "Axel", "Billy", "Charlie", "Peter", "Marie", "Paul"]

data = {'id':  np.arange(10000, 20000, 1).tolist(),
        'name': [random.choice(names) for _ in range(10000)],
        'age': np.random.randint(18, 67, size=10000),
        }

df = pd.DataFrame (data, columns = ['id', 'name', 'age'])
df

Something like this:

    id  name    age
0   10000   Charlie 45
1   10001   Peter   36
2   10002   Billy   34
3   10003   Axel    62
4   10004   Paul    20
... ... ... ...

This is an example list of dictionaries to be used to update the DataFrame with:

message_list = [
     {
    "id": 10002,
    "name": "Peter",
    "age": 65,
    },
     {
    "id": 10036,
    "name": "John",
    "age": 26,
    },
     {
    "id": 10789,
    "name": "Lisa",
    "age": 41,
    },
]

This is my current approach to update the name:

def update_df(df, message):
    df.loc[df.id == message["id"], 'name'] = message['name']

%%time
[update_df(df, message) for message in message_list]

CPU times: user 5.79 ms, sys: 494 µs, total: 6.29 ms
Wall time: 5.95 ms

Is there a faster way of doing this kind of DataFrame update? Maybe replacing the list comprehension with a more vectorized approach?

Answer 1

You could use .map on the series (from dictionary) and fillna with existing values.

In [260]: mapper = {d['id']:d['name'] for d in message_list}

In [261]: df['name'] = df['id'].map(mapper).fillna(df['name'])

In [262]: df
Out[262]: 
      id     name  age
0  10000  Charlie   45
1  10001    Peter   36
2  10002    Peter   34
3  10003     Axel   62
4  10004     Paul   20

---

In [269]: mapper
Out[269]: {10002: 'Peter', 10036: 'John', 10789: 'Lisa'}