有沒有辦法在 Python 中更有效（更快）地將批量 `yaml` 文件讀入 pandas `dataframe`

Question

我想從一個目錄中讀取幾個yaml文件到 pandas dataframe並將它們連接成一個大 Z388444BA115217A 該目錄包含7470個文件。

%%time
import pandas as pd
import glob

path = r'../input/cricsheet-a-retrosheet-for-cricket/all' # use your path
all_files = glob.glob(path + "/*.yaml")

li = []

for filename in all_files:
    with open(filename,'r') as fh:
        df = pd.json_normalize(yaml.load(fh.read()))
    li.append(df)

frame = pd.concat(li, axis=0, ignore_index=True)

CPU times: user 1h 15min 38s, sys: 8.8 s, total: 1h 15min 47s
Wall time: 1h 16min 44s

代碼運行時間超過一個多小時

有沒有辦法更有效地將批量yaml文件讀取到 pandas dataframe ？

壓縮樣本數據集

樣本數據集

Answer 1

step1：將yaml轉換為json文件，使用多進程

import os
from datetime import datetime, timedelta
from pandas import json_normalize
import pandas as pd
import numpy as np
import yaml

import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
ch = logging.StreamHandler()
ch.setLevel(logging.INFO)
formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
ch.setFormatter(formatter)
logger.addHandler(ch)

# yaml file path
os.chdir('~/Downloads/all')
yaml_file_list = os.listdir('.')
yaml_file_list =  [i for i in yaml_file_list if i.endswith('yaml')]

if not os.path.exists('output'):
    os.mkdir('output')
    
def yaml2json(file, cnt = 1):
    file_json = f'output/{file}.json'
    if os.path.exists(file_json):
        return
    # read yaml and convert to dict
    with open(file,'r') as fh:
        data = yaml.load(fh.read(), Loader=yaml.BaseLoader)
    
    # convert to json file
    data_str = json.dumps(data, ensure_ascii=False) + '\n'
    with open(file_json, 'w') as fw:
        fw.write(data_str)
    logging.info(f'[{cnt}] {file_json}')

file = yaml_file_list[0]
yaml2json(file)

# muti-Process to handle file to json
from concurrent.futures import ProcessPoolExecutor
####################
workers = 8
pool_list = yaml_file_list
pool_func = yaml2json
####################
total_count = len(pool_list)
with ProcessPoolExecutor(max_workers=workers) as executor:
    futures = [executor.submit(pool_func, param, total_count-n) 
                        for n, param in enumerate(pool_list)
            ]
  
# 2020-12-29 14:29:19,648 - INFO - [7468] output/1163055.yaml.json   
# 2020-12-29 14:32:07,597 - INFO - [6466] output/640941.yaml.json

# macbook 15' 2015
# 2.2 GHz Intel Core i7
# 16 GB 1600 MHz DDR3
# 1000 -> 14:29:19,648 -> 14:32:07,597 -> 3min
# 7400 ~ 25min

step2：將json文件合並為一個文件

os.chdir('~/Downloads/all/output/')

# merge file use bash cmd cat
# !cat *.json > yaml-all-json
# ipython
pycmd = lambda cmd: get_ipython().system(cmd)
cmd = 'cat *.json > yaml-all-json'
# pycmd(cmd)

step3：讀取json文件

# read file
# 1478 lines ->  4.37s
file = 'yaml-all-json'
df = pd.read_csv(file, sep='\n', header=None)[0]
obj = df.map(json.loads)
data_list = obj.tolist()
df_data = pd.DataFrame(data_list) # or use json_normalize to parse json data

df_data
#   meta    info    innings
# 0 {'data_version': '0.9', 'created': '2016-12-05...   {'dates': ['2016-11-24', '2016-11-25', '2016-1...   [{'1st innings': {'team': 'South Africa', 'dec...
# 1 {'data_version': '0.9', 'created': '2016-12-21...   {'city': 'Brisbane', 'dates': ['2016-12-15', '...   [{'1st innings': {'team': 'Australia', 'delive...
# 2 {'data_version': '0.9', 'created': '2016-10-21...   {'city': 'Port Moresby', 'dates': ['2016-10-16...   [{'1st innings': {'team': 'Papua New Guinea', ...
# 3 {'data_version': '0.9', 'created': '2016-09-14...   {'city': 'Edinburgh', 'dates': ['2016-09-10'],...   [{'1st innings': {'team': 'Scotland', 'deliver...
# 4 {'data_version': '0.9', 'created': '2016-09-12...   {'city': 'Londonderry', 'dates': ['2016-09-05'...   [{'1st innings': {'team': 'Hong Kong', 'delive..

Answer 2

如果您想避免進入並行計算的細節， Dask是一個很棒的 package。 它確實是為具有許多 CPU 的機器上的分布式計算而設計的，但我發現即使您只是將它用於一台機器上的多線程或多個進程，它的語法也很方便。

這是一些將 100 個 yaml 文件加載到 memory 中的代碼，首先不使用 Dask：

import glob
import yaml

path = r'all'  # local folder
all_files = glob.glob(path + "/*.yaml")

def load_yaml_file(filename):
    with open(filename, 'r') as fh:
        d = yaml.safe_load(fh.read())
    return d

n = 100
results = []
for filename in all_files[:n]:
    d = load_yaml_file(filename)
    results.append(d)
assert len(results) == n

然后，使用 Dask：

import dask

n = 100
lazy_results = []
for filename in all_files[:n]:
    d = dask.delayed(load_yaml_file)(filename)
    lazy_results.append(d)

results = dask.compute(*lazy_results, scheduler='processes')
assert len(results) == n

我在具有四核處理器的機器上對上述兩項進行了計時，發現使用 Dask 大約需要 19 秒（掛牆時間），而沒有使用 Dask 大約需要 1 分鍾多一點（大約 3.1 倍加速）。