读取一个大的 csv 文件然后将其拆分导致 OOM 错误

Question

Hi I'm creating a GLUE job that will read the csv file then split it via a particular column, unfortunately it's causing an OOM(Out of Memory) error. Please see code below

import pandas as pd
import numpy as np
from datetime import datetime, timedelta
import boto3


#get date 
Current_Date = datetime.now() - timedelta(days=1)
now = Current_Date.strftime('%Y-%m-%d')

#get date
Previous_Date = datetime.now() - timedelta(days=2)
prev = Previous_Date.strftime('%Y-%m-%d')

#read csv file that contain today's date
filepath = "s3://bucket/file"+now+".csv.gz"

data = pd.read_csv(filepath, sep='|', header=None,compression='gzip') 

#   count no. of loops
loop = 0
for i, x in data.groupby(data[10].str.slice(0,10)):
    loop += 1

# if no. of distinct values of column 10 (last_update) is greater than or equal to 7
if loop >= 7:
    #run loop for the dataframe and split by distinct values of column 10 (last_update)
    for i, x in data.groupby(data[10].str.slice(0, 10)):
        x.to_csv("s3://bucket/file.csv.gz".format(i.lower()),header=None,compression='gzip')

#if no. of distinct values of column 10 (last_update) is less than 7
#filter dateframe (current date and previous date); new dataframe is created
else:
    d = data[(data[10].str.slice(0,10)==prev)|(data[10].str.slice(0,10)==now)]
#run loop for the filtered data frame and split by distinct values of column 10 (last_update)
 for i, x in d.groupby(d[10].str.slice(0, 10)):
        x.to_csv("s3://bucket/file.csv.gz".format(i.lower()),header=None,compression='gzip')

SOLUTION - I resolved this problem by increasing the maximum capacity of the Glue Job

Answer 1

Not sure how big your file size is but if you split the file by chunks you should be able to avoid the error. We have successfully tested with a 2.5gb file using this method. Also if you are using the python shell remember to update your glue job maximum capacity to 1

data = pd.read_csv(filepath, chunksize=1000, iterator=True) 
for chunk in enumerate(data):
#Loop through the chunks and process the data