The loading of parquet files from HDFS is slower that the loading from S3. What can be the reasons for that?

Question

I have hundreds of parquet files in HDFS. I also have the same files in AWS S3. On a EMR cluster, I run a machine learning model, which can take its learning data from HDFS or from S3.

When it loads the data from HDFS, it takes a longer time than when the data is loaded from S3. Should not it be the opposite ? What can be the reason(s) for that?

Obviously, the hardware (the machines) used in the EMR cluster is the same for both cases.

Answer 1

S3 has problems when you load lots of files

Hadoop has problems when you use lots of small files (less than the 128 MB block size)

Assuming YARN settings are all the same, I would look at the IOPs rate to the HDFS volumes, and ensure your NodeManagers are running on the same machines as Datanodes