如何使用Pandas将文件写入S3

Question

I want to write a data frame column in .ann format to S3.

Right now I am using the following code to do that.

df['user_input'].to_csv(ann_file_path, header=None, index=None, sep=' ')

Where ann_file_path is the full path of the .ann file on the Server.

I am getting following error message:

[Errno 22] Invalid argument: 'https://s3-eu-west-1.amazonaws.com/bucket/sub_folder/somefile.ann'

Why am I getting that?

Also, do I need to use Boto3 to write or can I directly write the file on S3 with full path?

I can think of some authorization might be required for that but the error message seems different from something related to authorization.

Answer 1

I've resolved. We need AWS handshake using access_key_id and secret_key for AWS.

Get URL starting from the bucket name (not https:/...), hence get rid of whatever before it.

My URL: https://s3-eu-west-1.amazonaws.com/bucket/sub_folder/somefile.ann

Transformed to: bucket/sub_folder/somefile.ann

Code to do that: ann_file_path = ann_file_path.split('.com/', 1)[1]

Once I got ann_file_path , I used s3fs python library to upload the ann file to the server.

bytes_to_write = df['user_input'].to_csv(header=None, index=None).encode()
fs = s3fs.S3FileSystem(key=settings.AWS_ACCESS_KEY_ID, secret=settings.AWS_SECRET_ACCESS_KEY)
with fs.open(ann_file_path, 'wb') as f:
   f.write(bytes_to_write)