从HDFS加载实木复合地板文件比从S3加载慢。 可能是什么原因呢？

Question

I have hundreds of parquet files in HDFS. I also have the same files in AWS S3. On a EMR cluster, I run a machine learning model, which can take its learning data from HDFS or from S3.

When it loads the data from HDFS, it takes a longer time than when the data is loaded from S3. Should not it be the opposite ? What can be the reason(s) for that?

Obviously, the hardware (the machines) used in the EMR cluster is the same for both cases.

Answer 1

S3 has problems when you load lots of files

Hadoop has problems when you use lots of small files (less than the 128 MB block size)

Assuming YARN settings are all the same, I would look at the IOPs rate to the HDFS volumes, and ensure your NodeManagers are running on the same machines as Datanodes