[英]Writing files to local system with Spark in Cluster mode
I know this is a weird way of using Spark but I'm trying to save a dataframe to the local file system (not hdfs) using Spark even though I'm in cluster mode
. 我知道这是一种使用Spark的奇怪方式,但我正在尝试使用Spark将数据帧保存到本地文件系统(而不是hdfs),即使我处于
cluster mode
。 I know I can use client mode
but I do want to run in cluster mode
and don't care which node (out of 3) the application is going to run on as driver. 我知道我可以使用
client mode
但我确实希望以cluster mode
运行,而不关心应用程序将作为驱动程序运行的节点(3个中)。 The code below is the pseudo code of what I'm trying to do. 下面的代码是我正在尝试做的伪代码。
// create dataframe
val df = Seq(Foo("John", "Doe"), Foo("Jane", "Doe")).toDF()
// save it to the local file system using 'file://' because it defaults to hdfs://
df.coalesce(1).rdd.saveAsTextFile(s"file://path/to/file")
And this is how I'm submitting the spark application. 这就是我提交spark应用程序的方式。
spark-submit --class sample.HBaseSparkRSample --master yarn-cluster hbase-spark-r-sample-assembly-1.0.jar
This works fine if I'm in local mode
but doesn't in yarn-cluster mode
. 如果我处于
local mode
但不在yarn-cluster mode
这可以正常工作。
For example, java.io.IOException: Mkdirs failed to create file
occurs with the above code. 例如,
java.io.IOException: Mkdirs failed to create file
使用上面的代码java.io.IOException: Mkdirs failed to create file
。
I've changed the df.coalesce(1)
part to df.collect
and attempted to save a file using plain Scala but it ended up with a Permission denied
. 我已经将
df.coalesce(1)
部分更改为df.collect
并尝试使用普通Scala保存文件,但最终得到了Permission denied
。
I've also tried: 我也尝试过:
spark-submit
with root
user root
用户spark-submit
chown
ed yarn:yarn
, yarn:hadoop
, spark:spark
chown
ed yarn:yarn
, yarn:hadoop
, spark:spark
chmod 777
to related directories chmod 777
给相关目录 but no luck. 但没有运气。
I'm assuming this has to do something with clusters
, drivers and executors
, and the user
who's trying to write to the local file system but am pretty much stuck in solving this problem by myself. 我假设这必须对
clusters
, drivers and executors
尝试写入本地文件系统的user
drivers and executors
某些user
,但我自己一直坚持解决此问题。
I'm using: 我正在使用:
Any support is welcome and thanks in advance. 欢迎任何支持,并提前感谢。
Some articles I've tried: 我试过的一些文章:
chmod
didn't help me chmod
没有帮助我 This is the Exception I get. 这是我得到的例外。
java.io.IOException: Mkdirs failed to create file:/home/foo/work/rhbase/r/input/input.csv/_temporary/0/_temporary/attempt_201611242024_0000_m_000000_0 (exists=false, cwd=file:/yarn/nm/usercache/foo/appcache/application_1478068613528_0143/container_e87_1478068613528_0143_01_000001)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:449)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:435)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:920)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:813)
at org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:135)
at org.apache.spark.SparkHadoopWriter.open(SparkHadoopWriter.scala:91)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13.apply(PairRDDFunctions.scala:1193)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13.apply(PairRDDFunctions.scala:1185)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
16/11/24 20:24:12 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.io.IOException: Mkdirs failed to create file:/home/foo/work/rhbase/r/input/input.csv/_temporary/0/_temporary/attempt_201611242024_0000_m_000000_0 (exists=false, cwd=file:/yarn/nm/usercache/foo/appcache/application_1478068613528_0143/container_e87_1478068613528_0143_01_000001)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:449)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:435)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:920)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:813)
at org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:135)
at org.apache.spark.SparkHadoopWriter.open(SparkHadoopWriter.scala:91)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13.apply(PairRDDFunctions.scala:1193)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13.apply(PairRDDFunctions.scala:1185)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
I'm going to answer my own question because eventually, none of the answers didn't seem to solve my problem. 我将回答我自己的问题,因为最终,没有一个答案似乎没有解决我的问题。 None the less, thanks for all the answers and pointing me to alternatives I can check.
尽管如此,感谢所有答案并指出我可以检查的替代方案。
I think @Ricardo was the closest in mentioning the user of the Spark application. 我认为@Ricardo最近提到了Spark应用程序的用户。 I checked
whoami
with Process("whoami")
and the user was yarn
. 我用
Process("whoami")
检查了whoami
,用户是yarn
。 The problem was probably that I tried to output to /home/foo/work/rhbase/r/input/input.csv
and although /home/foo/work/rhbase
was owned by yarn:yarn
, /home/foo
was owned by foo:foo
. 问题可能是我试图输出到
/home/foo/work/rhbase/r/input/input.csv
,虽然/home/foo/work/rhbase
由yarn:yarn
拥有yarn:yarn
, /home/foo
归foo:foo
。 I haven't checked in detail but this may have been the cause of this permission
problem. 我没有详细检查,但这可能是此
permission
问题的原因。
When I hit pwd
in my Spark application with Process("pwd")
, it output /yarn/path/to/somewhere
. 当我使用
Process("pwd")
在我的Spark应用程序中点击pwd
时,它输出/yarn/path/to/somewhere
。 So I decided to output my file to /yarn/input.csv
and it was successful despite in cluster mode
. 所以我决定将我的文件输出到
/yarn/input.csv
,尽管在cluster mode
它仍然成功。
I probably can conclude that this was just a simple permission issue. 我可能会得出结论,这只是一个简单的权限问题。 Any further solution would be welcome but for now, this was the way how I solved this question.
任何进一步的解决方案都会受到欢迎,但就目前而言,这就是我解决这个问题的方式。
Use forEachPartition method, and then for each partition get file system object and write one by one record to it, below is the sample code here i am writing to hdfs, instead you can use local file system as well 使用forEachPartition方法,然后为每个分区获取文件系统对象并逐一写入记录,下面是我在写hdfs的示例代码,而不是你可以使用本地文件系统
Dataset<String> ds=....
ds.toJavaRdd.foreachPartition(new VoidFunction<Iterator<String>>() {
@Override
public void call(Iterator<String> iterator) throws Exception {
final FileSystem hdfsFileSystem = FileSystem.get(URI.create(finalOutPathLocation), hadoopConf);
final FSDataOutputStream fsDataOutPutStream = hdfsFileSystem.exists(finalOutPath)
? hdfsFileSystem.append(finalOutPath) : hdfsFileSystem.create(finalOutPath);
long processedRecCtr = 0;
long failedRecsCtr = 0;
while (iterator.hasNext()) {
try {
fsDataOutPutStream.writeUTF(iterator.next);
} catch (Exception e) {
failedRecsCtr++;
}
if (processedRecCtr % 3000 == 0) {
LOGGER.info("Flushing Records");
fsDataOutPutStream.flush();
}
}
fsDataOutPutStream.close();
}
});
If you run the job as yarn-cluster mode
, the driver will be running in any of the machine which is managed by YARN, so if saveAsTextFile
has local file path, then it will store the output in any of the machine where driver is running. 如果您以
yarn-cluster mode
运行作业,驱动程序将在由YARN管理的任何机器中运行,因此如果saveAsTextFile
具有本地文件路径,则它将输出存储在运行驱动程序的任何机器中。
Try running the job as yarn-client mode
so the driver runs in the client machine 尝试将作业作为
yarn-client mode
运行,以便驱动程序在客户端计算机中运行
Please refer to the spark documentation to understand the use of --master
option in spark-submit
. 请参阅spark文档以了解
spark-submit
使用--master
选项。
--master local
is supposed to be used when running locally. --master local
运行时应该使用--master local
。
--master yarn --deploy-mode cluster
is supposed to be used when actually running on a yarn cluster. --master yarn --deploy-mode cluster
应该在纱线群集上实际运行时使用。
Check if you are trying to run/write the file with a user other than the Spark service. 检查您是否尝试使用Spark服务以外的用户运行/写入文件。 On that situation you can solve the permission issue by presetting the directory ACLs.
在这种情况下,您可以通过预设目录ACL来解决权限问题。 Example:
例:
setfacl -d -m group:spark:rwx /path/to/
(modify "spark" to your user group trying to write the file) (修改“spark”到试图写文件的用户组)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.