从 pyspark 访问 s3 时，亚马逊存储桶的证书不匹配

Question

I have EC2 instance where I'm trying to configure PySpark to read from S3. I set a full access IAM role to EC2 instance and used the following packages in spark:

com.amazonaws:aws-java-sdk-bundle:1.11.563,org.apache.hadoop:hadoop-aws:3.3.1

However, I'm getting some new error, and I'm not sure what does it mean:

: org.apache.hadoop.fs.s3a.AWSClientIOException: getFileStatus on s3a://bucket_name.stuff/mycsv.csv: com.amazonaws.SdkClientException: Unable to execute HTTP request: Certificate for <bucket_name.stuff.s3.amazonaws.com> doesn't match any of the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com]

Answer 1

So the issue turned out to be a version mismatch between pyspark, hadoop-aws and java-sdk (I was getting all kind of different errors until I found a proper version setup) The combination that worked for me was:

pyspark 3.0.0
org.apache.hadoop:hadoop-aws:2.7.2
com.amazonaws:aws-java-sdk-pom:1.11.34