[英]Writing file using FileSystem to S3 (Scala)
I'm using scala , and trying to write file with string content, to S3. 我正在使用scala,并尝试将具有字符串内容的文件写入S3。 I've tried to do that with FileSystem , but I getting an error of: "Wrong FS: s3a"
我尝试使用FileSystem做到这一点,但出现错误:“错误的FS:s3a”
val content = "blabla"
val fs = FileSystem.get(spark.sparkContext.hadoopConfiguration)
val s3Path: Path = new Path("s3a://bucket/ha/fileTest.txt")
val localPath= new Path("/tmp/fileTest.txt")
val os = fs.create(localPath)
os.write(content.getBytes)
fs.copyFromLocalFile(localPath,s3Path)
and i'm getting an error: 我收到一个错误:
java.lang.IllegalArgumentException: Wrong FS: s3a://...txt, expected: file:///
What is wrong? 怎么了?
Thanks!! 谢谢!!
you need to ask for the specific filesystem for that scheme, then you can create a text file directly on the remote system. 您需要为该方案要求特定的文件系统,然后可以直接在远程系统上创建文本文件。
val s3Path: Path = new Path("s3a://bucket/ha/fileTest.txt")
val fs = s3Path.getFilesystem(spark.sparkContext.hadoopConfiguration)
val os = fs.create(s3Path, true)
os.write("hi".getBytes)
os.close
There's no need to write locally and upload; 无需本地编写和上传; the s3a connector will buffer and upload as needed
s3a连接器将根据需要缓冲并上传
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.