如何使用Luigi执行Spark任务？

Question

我运行了很多Spark作业，我想使用Luigi进行协调。 但是，我收到一条错误消息，我不知道该如何解决。

这是我的基本脚本：

import luigi
from luigi.contrib.spark import PySparkTask
import pandas as pd

class rddTest(PySparkTask):

    def input(self):
        d = [1, 2, 3, 4, 5]
        return d

    def output(self):
        return luigi.LocalTarget("rdd.csv")

    def main(self, sc, *args):

        rdd1 = sc.parallelize(self.input())
        df = rdd1.toPandas()
        df.to_csv(self.output().path)

if __name__ == "__main__":
    luigi.run()

我的luigi.cfg在下面。 我在一家大公司工作，刚刚被批准使用Luigi，因此怀疑是否在集群上设置了中央调度程序。

[core]
default-scheduler-host:name_of_your_luigi_server
python-home-dir:$PYTHON_HOME

[spark]
spark-submit:$SPARK_HOME/bin/spark-submit
hadoop-conf-dir:$HADOOP_HOME
yarn-conf-dir:$YARN_HOME
master:yarn-client
num-executors:10
executor-memory: 64g
executor-memory: 140g

这样，我希望在启动脚本的同一目录中看到一个名为rdd.csv的csv文件，但不是。

使用此启动cmd

python sparkTest.py --local-scheduler rddTest

我得到错误

DEBUG: Checking if rddTest() is complete
INFO: Informed scheduler that task   rddTest__99914b932b   has status   PENDING
INFO: Done scheduling tasks
INFO: Running Worker with 1 processes
DEBUG: Asking scheduler for work...
DEBUG: Pending tasks: 1
INFO: [pid 50029] Worker Worker(salt=326401154, workers=1, host=hpchdp2e, username=<username>, pid=50029) running   rddTest()
INFO: Running command: $SPARK_HOME/bin/spark-submit --master yarn-client --name rddTest --executor-memory 140g --num-executors 10 /s/anaconda/users/<username>/miniconda2/lib/python2.7/site-packages/luigi/contrib/pyspark_runner.py /tmp/rddTestqRsQ4F/rddTest.pickle
ERROR: [pid 50029] Worker Worker(salt=326401154, workers=1, host=hpchdp2e, username=<username>, pid=50029) failed    rddTest()
Traceback (most recent call last):
  File "/s/anaconda/users/<username>/miniconda2/lib/python2.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/s/anaconda/users/<username>/miniconda2/lib/python2.7/site-packages/luigi/worker.py", line 141, in _run_get_new_deps
    task_gen = self.task.run()
  File "/s/anaconda/users/<username>/miniconda2/lib/python2.7/site-packages/luigi/contrib/spark.py", line 309, in run
    super(PySparkTask, self).run()
  File "/s/anaconda/users/<username>/miniconda2/lib/python2.7/site-packages/luigi/contrib/spark.py", line 66, in run
    super(SparkSubmitTask, self).run()
  File "/s/anaconda/users/<username>/miniconda2/lib/python2.7/site-packages/luigi/contrib/external_program.py", line 134, in run
    with self._proc_with_tracking_url_context(proc_args=args, proc_kwargs=kwargs) as proc:
  File "/s/anaconda/users/<username>/miniconda2/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/s/anaconda/users/<username>/miniconda2/lib/python2.7/site-packages/luigi/contrib/external_program.py", line 168, in _proc_with_tracking_url_context
    main_proc = subprocess.Popen(proc_args, **proc_kwargs)
  File "/s/anaconda/users/<username>/miniconda2/lib/python2.7/subprocess.py", line 394, in __init__
    errread, errwrite)
  File "/s/anaconda/users/<username>/miniconda2/lib/python2.7/subprocess.py", line 1047, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

任何帮助或见解表示赞赏！

Answer 1

您似乎错误地评估了luigi.cfg文件中的环境变量。 在这里（ https://luigi.readthedocs.io/en/stable/configuration.html ）提到要在luigi.cfg使用环境变量，您需要使用${VARIABLE}才能正确使用它们，因此您的配置将看起来像：

[core]
default-scheduler-host:name_of_your_luigi_server
python-home-dir:${PYTHON_HOME}

[spark]
spark-submit:${SPARK_HOME}/bin/spark-submit
hadoop-conf-dir:${HADOOP_HOME}
yarn-conf-dir:${YARN_HOME}
master:yarn-client
num-executors:10
executor-memory: 64g
executor-memory: 140g