简体繁体 English

Spark提交的HDFS路径和YARN上的Flink

[英]HDFS Path for Spark Submit and Flink on YARN

原文 2015-06-05 13:47:32 5 1 java/ hadoop/ apache-spark/ hdfs/ apache-flink

i work with cloudera live vm, there i have a hadoop and spral standalone cluster. 我使用cloudera live vm，我有一个hadoop和spral独立集群。 now i want submit my jobs with spark submit and flink run scripts. 现在我想用spark提交和flink运行脚本提交我的工作。 this works, too. 这也有效。 but my apps can find the path to input and outputs files in the hdfs. 但我的应用程序可以在hdfs中找到输入和输出文件的路径。 i set the path like: hdfs://127.0.0.1:50010/user/cloudera/outputs the information from the port i get from this: 我设置路径如：hdfs：//127.0.0.1：50010 / user / cloudera /输出来自我得到的端口的信息： 在此输入图像描述

how to set the path for the hdfs in java?? 如何在java中设置hdfs的路径？

best regards, paul 最好的问候，保罗

1 个解决方案

You don't have to set the path to the DataNode host. 您不必设置DataNode主机的路径。 On the "Overview" page, you'll see the connection information for the NameNode, and that is also where you have to connect to. 在“概述”页面上，您将看到NameNode的连接信息，这也是您必须连接的位置。

Spark 在 Yarn Cluster 模式下提交，并将配置文件放入 HDFS 问题 - Spark submit on Yarn Cluster mode with config file put into HDFS issue

在 Yarn 上运行 Spark 作业时，HDFS Excel Rows 减少了 - HDFS Excel Rows got decreased when running the spark job on Yarn

从本地 spark-submit 检查远程 HDFS 上是否存在文件 - Check if file exists on remote HDFS from local spark-submit

使用--jars的spark-submit yarn-cluster不起作用？ - spark-submit yarn-cluster with --jars does not work?

为什么 spark-submit 无法部署到 YARN？ - Why does spark-submit fail to deploy to YARN?

SparkLauncher用用户作为蜂巢的yarn-client运行spark-submit - SparkLauncher Run spark-submit with yarn-client with user as hive

如何通过Yarn，Hadoop提交Spark Scala作业 - How to submit a Spark scala job over Yarn, Hadoop

不使用spark-submit.sh时，Spark如何知道纱线资源管理器在哪里运行？ - How does Spark know where the Yarn Resource Manager is running when not using spark-submit.sh?

Spark提交作业在集群模式下失败，但在Java中可用于本地HDFS中的copyToLocal - Spark submit job fails for cluster mode but works in local for copyToLocal from HDFS in java

pyspark hiveContext 错误，同时执行 spark-submit 应用程序到纱线和远程 CDH kerberized env - pyspark hiveContext error while executing spark-submit application to yarn and remote CDH kerberized env

暂无

暂无

声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 Spark 在 Yarn Cluster 模式下提交，并将配置文件放入 HDFS 问题 - Spark submit on Yarn Cluster mode with config file put into HDFS issue 在 Yarn 上运行 Spark 作业时，HDFS Excel Rows 减少了 - HDFS Excel Rows got decreased when running the spark job on Yarn 从本地 spark-submit 检查远程 HDFS 上是否存在文件 - Check if file exists on remote HDFS from local spark-submit 使用--jars的spark-submit yarn-cluster不起作用？ - spark-submit yarn-cluster with --jars does not work? 为什么 spark-submit 无法部署到 YARN？ - Why does spark-submit fail to deploy to YARN? SparkLauncher用用户作为蜂巢的yarn-client运行spark-submit - SparkLauncher Run spark-submit with yarn-client with user as hive 如何通过Yarn，Hadoop提交Spark Scala作业 - How to submit a Spark scala job over Yarn, Hadoop 不使用spark-submit.sh时，Spark如何知道纱线资源管理器在哪里运行？ - How does Spark know where the Yarn Resource Manager is running when not using spark-submit.sh? Spark提交作业在集群模式下失败，但在Java中可用于本地HDFS中的copyToLocal - Spark submit job fails for cluster mode but works in local for copyToLocal from HDFS in java pyspark hiveContext 错误，同时执行 spark-submit 应用程序到纱线和远程 CDH kerberized env - pyspark hiveContext error while executing spark-submit application to yarn and remote CDH kerberized env

相关标签

粤ICP备18138465号 © 2020-2024 STACKOOM.COM