Apache Airflow - Spark 提交失败 - 使用主“yarn-client”运行时，必须在环境中设置 HADOOP_CONF_DIR 或 YARN_CONF_DIR

Question

我是 Spark 和 Airflow 的新手，正在尝试创建一个在 pyspark 中运行 spark 提交作业的 DAG。

在我的 Ubuntu 系统中，我创建了一个名为“hadoopusr”的用户，我通过它手动运行我的 spark 提交。 所有环境变量都在该用户下的/.bashrc中设置。

当我从终端手动运行 spark-submit 时，作业运行成功。

我创建了一个示例 DAG 文件，如下所示：

"""
Apache Airflow DAG Script: Ingestion MVP basic version
"""
#imports
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from datetime import datetime, timedelta
import os
import sys
#Paths
os.environ['SPARK_HOME'] = '/usr/local/spark/spark-2.4.4-bin-hadoop2.7'
sys.path.append(os.path.join(os.environ['SPARK_HOME'], 'bin'))
#Parameters
filename="CDC_DAY1"
file_extension="csv"
batch_id="20191203"
## Define the DAG object
default_args = {
    'owner': 'hadoopusr',
    'depends_on_past': False,
    'start_date': datetime(2019, 12, 3),
    'retries': 5,
    'retry_delay': timedelta(minutes=1),
}
dag = DAG('MVP_BASIC_2', default_args=default_args, schedule_interval=timedelta(1))
#List of tasks
LandingToRaw = BashOperator(
    task_id='Landing_to_Raw',
    bash_command="spark-submit --master yarn-client /home/soham/Documents/mvp_landing_raw_1.py "+filename+" "+file_extension+" "+batch_id,
    dag=dag,
    run_as_user='hadoopusr')

RawToStaging = BashOperator(
    task_id='Raw_to_Staging',
    bash_command="spark-submit --master yarn-client /home/soham/Documents/mvp_raw_staging_1.py "+filename+" "+file_extension+" "+batch_id+" FULL emp_id",
    dag=dag,
    run_as_user='hadoopusr')

#Dependencies
LandingToRaw >> RawToStaging

当我在同一用户 (hadoopusr) 下使用以下命令测试 DAG 的第一个任务时，它会抛出如下异常：

Running command: spark-submit --master yarn-client /home/soham/Documents/mvp_landing_raw_1.py CDC_DAY1 csv 20191203
[2019-12-04 19:27:52,527] {bash_operator.py:124} INFO - Output:
[2019-12-04 19:27:54,937] {bash_operator.py:128} INFO - Exception in thread "main" org.apache.spark.SparkException: When running with master 'yarn-client' either HADOOP_CONF_DIR or YARN_CONF_DIR must be set in the environment.
[2019-12-04 19:27:54,938] {bash_operator.py:128} INFO -     at org.apache.spark.deploy.SparkSubmitArguments.error(SparkSubmitArguments.scala:657)
[2019-12-04 19:27:54,938] {bash_operator.py:128} INFO -     at org.apache.spark.deploy.SparkSubmitArguments.validateSubmitArguments(SparkSubmitArguments.scala:290)
[2019-12-04 19:27:54,938] {bash_operator.py:128} INFO -     at org.apache.spark.deploy.SparkSubmitArguments.validateArguments(SparkSubmitArguments.scala:251)
[2019-12-04 19:27:54,938] {bash_operator.py:128} INFO -     at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:120)
[2019-12-04 19:27:54,939] {bash_operator.py:128} INFO -     at org.apache.spark.deploy.SparkSubmit$$anon$2$$anon$1.<init>(SparkSubmit.scala:907)
[2019-12-04 19:27:54,939] {bash_operator.py:128} INFO -     at org.apache.spark.deploy.SparkSubmit$$anon$2.parseArguments(SparkSubmit.scala:907)
[2019-12-04 19:27:54,939] {bash_operator.py:128} INFO -     at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:81)
[2019-12-04 19:27:54,939] {bash_operator.py:128} INFO -     at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
[2019-12-04 19:27:54,940] {bash_operator.py:128} INFO -     at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
[2019-12-04 19:27:54,940] {bash_operator.py:128} INFO -     at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
[2019-12-04 19:27:54,973] {bash_operator.py:132} INFO - Command exited with return code 1
[2019-12-04 19:27:54,987] {taskinstance.py:1058} ERROR - Bash command failed
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/airflow/models/taskinstance.py", line 930, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/usr/local/lib/python3.6/dist-packages/airflow/operators/bash_operator.py", line 136, in execute
    raise AirflowException("Bash command failed")
airflow.exceptions.AirflowException: Bash command failed
[2019-12-04 19:27:54,990] {taskinstance.py:1081} INFO - Marking task as UP_FOR_RETRY
Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 37, in <module>
    args.func(args)
  File "/usr/local/lib/python3.6/dist-packages/airflow/utils/cli.py", line 74, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/airflow/bin/cli.py", line 688, in test
    ti.run(ignore_task_deps=True, ignore_ti_state=True, test_mode=True)
  File "/usr/local/lib/python3.6/dist-packages/airflow/utils/db.py", line 74, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/airflow/models/taskinstance.py", line 1020, in run
    session=session)
  File "/usr/local/lib/python3.6/dist-packages/airflow/utils/db.py", line 70, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/airflow/models/taskinstance.py", line 930, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/usr/local/lib/python3.6/dist-packages/airflow/operators/bash_operator.py", line 136, in execute
    raise AirflowException("Bash command failed")
airflow.exceptions.AirflowException: Bash command failed

我会在配置或其他任何地方出错吗？

Answer 1

您正在尝试在yarn-client模式下运行spark，为此您需要配置YARN，如果您使用的是YARN，则需要将YARN_CONF_DIR设置为yarn配置的路径，您也可以使用spark的本地主模式运行- -主本地[num_of_threads/*]。

Answer 2

我在使用 Airflow 提交 Spark 作业时遇到了同样的问题。 我意识到问题出在airflow-webserver.service文件中的EnvironmentFile变量。

对我来说，在我将~/.bashrc上的所有变量（如HADOOP_HOME, HADOOP_CONF_DIR, YARN_HOME, SPARK_HOME, etc复制到<EnvironmentFile>路径后，这个问题得到了解决，在我的例子中，路径是/etc/sysconfig/airflow （这是默认值，你可以在这里查看https://raw.githubusercontent.com/apache/airflow/master/scripts/systemd/airflow-webserver.service ）。 airflow-scheduler.service中也存在相同的变量。 复制到etc/sysconfig/airflow后，我重新启动了airflow-webserver和airflow-scheduler服务。 Spark 作业从 Airflow 成功执行。

示例：如果在 bashrc 文件上有行export HADOOP_HOME=/opt/hadoop ，则只需要在 EnvironmentFile 上保留HADOOP_HOME=/opt/hadoop （无需导出）