Write parquet data to a file on S3 with scala
Question
有没有一种方法可以使用Scala在S3上快速稳定地写入镶木地板数据？
2 answers
solution1
0 2017-02-17 08:06:44
是的-使用Spark一旦拥有一个数据df.write.parquet("s3a://newfilename")就可以使用df.write.parquet("s3a://newfilename")
solution2
0 2017-02-18 15:06:50
you can work with s3a long haul from a standalone installation if you want to, though I'd recommend against using it as the direct destination of work because of performance issues. Better to output locally then copy it there (hadoop distcp does this).
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.
Related Question
Error writing parquet file in S3 with Apache Spark in Scala
loop a sequence of s3 parquet file path with same schema and save in a single dataframe in scala
Using Spark to write a parquet file to s3 over s3a is very slow
Slow performance reading parquet files in S3 with scala in Spark
Trying to read and write parquet files from s3 with local spark
s3 parquet write - too many partitions, slow writing
How to write constant value in parquet file using scala?
Strange error while writing parquet file to s3
Spark Streaming - Parquet file upload to S3 error
How to convert Parquet file to Protobuf and save it HDFS/AWS S3
Related Tags