如何加速這個 Pandas 數據幀操作？

Question

我正在與 Pandas 一起對新聞文章列表進行一些計算，即在按日期分組時獲取 NLP 數據的平均值，並按源輸出到 JS 圖表中。 對於 20k 條記錄，操作需要 2 到 3 秒。 如果可能的話，我想把它降到小於 0.5。 代碼是：

articles = [{'title': "article title", 'rounded_polarity': 63, 'rounded_subjectivity': 45, 'source_name': 'foxnews', 'day': '2020-01-11 00:00:00+00:00'}, ...]

def get_averages(articles):
    data_frame = DataFrame(articles)
    grouped_by_day = data_frame.groupby(['day']).mean()
    grouped_by_source = data_frame.groupby(['source_name']).mean()

    grouped_by_day_dict = grouped_by_day.to_dict()
    grouped_by_source_dict = grouped_by_source.to_dict()
    max_sentiments = grouped_by_source.idxmax().to_dict()
    min_sentiments = grouped_by_source.idxmin().to_dict()

    total_avg_subjectivity = statistics.mean([v for k, v in grouped_by_source_dict['rounded_subjectivity'].items()])
    total_avg_sentiment = statistics.mean([v for k, v in grouped_by_source_dict['rounded_polarity'].items()])

    return {
        'most_positive_source': max_sentiments['rounded_polarity'],
        'least_positive_source': min_sentiments['rounded_polarity'],
        'most_subjective_source': max_sentiments['rounded_subjectivity'],
        'least_subjective_source': min_sentiments['rounded_subjectivity'],
        'average_sentiment': total_avg_sentiment,
        'average_subjectivity': total_avg_subjectivity,
        'averages_by_day': grouped_by_day_dict,
        'earliest_publish_date': grouped_by_day.index.min(),
        'latest_publish_date': grouped_by_day.index.max()

我如何利用更多的 Pandas 內置功能來加快速度？

Answer 1

好的，我認為 pandas 和 numpy 的方法與您所做的非常相似，只需使用內置函數和方法：

import pandas as pd
import numpy as np

articles = [{'title': "article title", 'rounded_polarity': 63, 'rounded_subjectivity': 45, 'source_name': 'foxnews', 'day': '2020-01-11 00:00:00+00:00'}]

df = pd.DataFrame(articles)
grouped_by_day = df.groupby('day').mean()
grouped_by_source = df.groupby('source_name').mean()


max_sentiments = grouped_by_source.idxmax()
min_sentiments = grouped_by_source.idxmin()

total_avg = np.mean(grouped_by_source.to_numpy()) # equivalent to grouped_by_source.mean() if you don't want to add numpy dependency, however numpy is faster! 

result = {'most_positive_source': max_sentiments['rounded_polarity'],
          'least_positive_source': min_sentiments['rounded_polarity'],
          'most_subjective_source': max_sentiments['rounded_subjectivity'],
          'least_subjective_source': min_sentiments['rounded_subjectivity'],
          'average_sentiment': total_avg['rounded_polarity'],
          'average_subjectivity': total_avg['rounded_subjectivity'],
          'averages_by_day': grouped_by_day.to_dict(),
          'earliest_publish_date': grouped_by_day.index.min(),
          'latest_publish_date': grouped_by_day.index.max()}