memory error while writing a large dataframe to S3 AWS

Question

I have created a dataframe with the following shape using amazon sagemaker.

10612611 rows × 4 columns

All are numeric values. When I am trying to write this dataframe into my S3 bucket as follows, I get memory error.

bytes_to_write = df.to_csv(None).encode()
with s3.open('aws-athena-query-results-xxxxxxx/query_result/xx.csv','wb') as f:
    f.write(bytes_to_write)

MemoryError:

I am using ml.t2.medium for sagemaker instance.

Answer 1

I solved this issue by changing the instance type from ml.t2.medium to ml.t2.2xlarge and it worked perfectly.

The original issue was with the RAM of the instance type and not with S3.