我想知道 AWS EMR 发送大量 S3 model 文件的列表和头部请求是否正常

Question

I am using the AWS EMR Cluster service. It is a situation in which machine learning tasks such as spark-build are being performed by referring to the model file with the S3 Bucket between EMR Cluster use. I request a lot of head and list requests from S3, but I am wondering if it is normal for AWS EMR to send a lot of list and head requests to the S3 model file. Symptom: AWS EMR is about 2.7 million head and list requests per day to S3.

Answer 1

A lot of list/head requests get sent.

This is related to how directories are emulated on the hadoop/spark/hive S3 clients; every time a progress looks to see if there's a directory on a path it will issue a LIST request, maybe a HEAD request first (to see if its a file).

Then there's the listing of the contents, more LIST requests, and finally reading the files. There'll be one HEAD request on every open() call to verify the file exists and to determine how long it is.

Files are read with GET Requests. Every time there's a seek()/buffer read on the input stream and the data isn't in a buffer the client has to do one of

• read to the end of the current ranged get (assuming its a ranged GET), discarding the data, issue a new ranged GET
• abort() the HTTPS connection, negotiate a new one. Slow.

Overall then, a lot of IO, especially if the application is inefficient about caching the output of directory listings, whether files exist, doing needless checks before operations ( if fs.exists(path) fs.delete(path, false) ) and the like. If this is your code, try not to do that

(disclaimer: this is all guesses based on the experience of tuning the open source hive/spark apps to work through the S3A connector. I'm assuming the same for EMR)