AWS credentials not found when using spark/scal app to access s3

Question

I'm using a windows environment, hadoop is not installed, I have a hadoop folder with bin and the winutils, that is all.

I've set environment variables for both the id and secret as per the docs but constantly getting this exception:

Exception in thread "main" java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key must be specified as the username or password (respectively) of a s3n URL, or by setting the fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey properti

any idea how I can fix please?

I've confirmed the variables are set by echoing them but they're not getting picked up - thanks./

Answer 1

Have you tried this configuration?

val sc = new SparkContext(conf)
val hadoopConf = sc.hadoopConfiguration;
hadoopConf.set("fs.s3.impl", "org.apache.hadoop.fs.s3native.NativeS3FileSystem")
hadoopConf.set("fs.s3.awsAccessKeyId",myAccessKey)
hadoopConf.set("fs.s3.awsSecretAccessKey",mySecretKey)