Spark在S3上不输出.crc文件

Question

When I use spark locally, writing data on my local filesystem, it creates some usefull .crc file.

Using the same job on Aws EMR and writing on S3, the .crc files are not written.

Is this normal? Is there a way to force the writing of .crc files on S3?

Answer 1

those .crc files are just created by the the low level bits of the Hadoop FS binding so that it can identify when a block is corrupt, and, on HDFS, switch to another datanode's copy of the data for the read and kick off a re-replication of one of the good copies.

On S3, stopping corruption is left to AWS.

What you can get off S3 is the etag of a file, which is the md5sum on a small upload; on a multipart upload it is some other string, which again, changes when you upload it.

you can get at this value with the Hadoop 3.1+ version of the S3A connector, though it's off by default as distcp gets very confused when uploading from HDFS. For earlier versions, you can't get at it, nor does the aws s3 command show it. You'd have to try some other S3 libraries (it's just a HEAD request, after all)