Writing file using FileSystem to S3 (Scala)

Question

I'm using scala , and trying to write file with string content, to S3. I've tried to do that with FileSystem , but I getting an error of: "Wrong FS: s3a"

    val content = "blabla"
    val fs = FileSystem.get(spark.sparkContext.hadoopConfiguration)
    val s3Path: Path = new Path("s3a://bucket/ha/fileTest.txt")
    val localPath= new Path("/tmp/fileTest.txt")
    val os = fs.create(localPath)
    os.write(content.getBytes)
    fs.copyFromLocalFile(localPath,s3Path)

and i'm getting an error:

java.lang.IllegalArgumentException: Wrong FS: s3a://...txt, expected: file:///

What is wrong?

Thanks!!

Answer 1

you need to ask for the specific filesystem for that scheme, then you can create a text file directly on the remote system.

val s3Path: Path = new Path("s3a://bucket/ha/fileTest.txt")
val fs = s3Path.getFilesystem(spark.sparkContext.hadoopConfiguration)
val os = fs.create(s3Path, true)
os.write("hi".getBytes)
os.close

There's no need to write locally and upload; the s3a connector will buffer and upload as needed