Hadoop可访问S3文件，但Spark无法访问

Question

I am running Spark 1.4 along Hadoop 2.6 on a single EC2 machine. I configured the HADOOP_CLASSPATH and core-site.xml to get access to my S3 files.

While Hadoop is able to access the files in my bucket, spark-shell fails miserably throwing the following error:

 java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3native.NativeS3FileSystem not found

I tried adding the aws jars to the classpath, but nothing helps.

Anyone has any idea where this might be coming from?

Thanks!

Answer 1

You need to add two extra jar files into the class path

eg. in your spark-submit --jar=aws-java-sdk-1.7.4.jar:hadoop-aws-2.6.0.jar

or you can add into your config: eg. spark.executor.extraClassPath & spark.driver.extraClassPath

In addition, try to use "s3a://" which is a newer S3 library in Hadoop