使用 Scala 将文件写入 Amazon S3 AWS 存储

Question

this is part of my code:

   val file=new File("s3a://tracce/output/Tempi.txt") 
   val writer=new BufferedWriter(new FileWriter(file,true))
   writer.write("Execution Time trace "+count.toString+"x"+i.toString+": "+differenza.toString)
   writer.newLine()
   writer.flush()
   writer.close()

I need to write a new file into my bucket using Scala. I get this error:

Exception in thread "main" java.io.FileNotFoundException:     s3:/tracce/output/Tempi.txt (No such file or directory)

When I try to save it locally, it works:

val file=new File("./Tempi.txt")

How could I solve it? Thanks in advance

Answer 1

S3A being a Hadoop Filesystem client, you can't use the java File API to work with it.

1. Create a hadoop Configuration object with all your s3 credentials in them. If you are using Spark, you get that for free from SparkContext .
2. Use the Hadoop FS API to get the FS instance, create() to create a file in it.

import org.apache.hadoop.fs._
    import org.apache.hadoop.conf.Configuration
    val conf = new Configuration()
    conf.set("fs.s3a.access.key", "YOUR ACCESS KEY")
    conf.set("fs.s3a.secret.key", "YOUR SECRET KEY")

    val dest = new Path("s3a://bucket_name/output-path/file.txt")
    val fs = dest.getFileSystem(conf)
    val out = fs.create(dest, true)
    out.write( /* your string as bytes */ )
    out.close()

Answer 2

You can now write it directly using the AWS Java SDK. Here is the example;

import com.amazonaws.services.s3.{AmazonS3, AmazonS3ClientBuilder}

val s3Client: AmazonS3 = AmazonS3ClientBuilder.defaultClient()

val bucketName = "bucket-name"
val objectKey = "object-key"
val objectContent = "This is the file content."

s3Client.putObject(bucketName, objectKey, objectContent)

Answer 3

S3 being an object store, you cannot write strings. You can probably write the strings to a local file and periodically upload the file to S3.