Apache Spark S3错误

Question

I am trying to connect amazon s3 to Sparkstreaming . I am running code on my local machine and trying to stream from s3 to Spark and I got below error:

java.io.IOException: No FileSystem for scheme: s3n

Can you please help me in solving the same?

Answer 1

You can solve it by specifying the implementation of s3n scheme in the hadoop configuration of your spark context:

sparkContext.hadoopConfiguration.set("fs.s3.impl", "org.apache.hadoop.fs.s3native.NativeS3FileSystem")

In order to access s3 you may also need to specify AWS credentials:

sparkContext.hadoopConfiguration.set("fs.s3n.awsAccessKeyId", "***")
sparkContext.hadoopConfiguration.set("fs.s3n.awsSecretAccessKey", "***")

Then you can create your StreamingContext in the following way:

val ssc = new StreamingContext(sc, Seconds(1))

You may want to try accessing s3 through s3a:// instead of s3n:// which uses aws-sdk library instead of jets3t to access the files.