[英]apache-airflow systemd file using a conda env
I'm trying to run apache-airflow
on a Ubuntu 16.04 file, using systemd.我正在尝试使用 systemd 在 Ubuntu 16.04 文件上运行
apache-airflow
airflow。 I roughly followed this tutorial and installed/setup the following:我大致遵循了本教程并安装/设置了以下内容:
sudo apt-get install gcc
)sudo apt-get install gcc
)yml
file of the tutorial yml
文件Within the following conda environment:在以下 conda 环境中:
When I test Airflow, everything works fine:当我测试 Airflow 时,一切正常:
airflow webserver --port 8080
But whenever I try to launch airflow using a systemd file, it fails.但是每当我尝试使用 systemd 文件启动气流时,它都会失败。 The systemd file makes use of the conda environment, as far as I understand correctly.
据我所知,systemd 文件使用了 conda 环境。 My systemd file looks as follows:
我的 systemd 文件如下所示:
[Unit]
Description=Airflow webserver daemon
[Service]
User=ubuntu
Group=ubuntu
Type=simple
ExecStart=/home/ubuntu/miniconda2/envs/airflow-tutorial/bin/airflow webserver --port 8080
Restart=on-failure
RestartSec=5s
PrivateTmp=true
[Install]
WantedBy=multi-user.target
When I start/enable the systemd daemon, status returns the following error:当我启动/启用 systemd 守护进程时,状态返回以下错误:
airflow-webserver.service - Airflow webserver daemon
Loaded: loaded (/etc/systemd/system/airflow-webserver.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Thu 2018-09-13 08:59:00 UTC; 1s ago
Process: 18410 ExecStart=/home/ubuntu/miniconda2/envs/airflow-tutorial/bin/airflow webserver --port 8080 (code=exited, status=1/FAILURE)
Main PID: 18410 (code=exited, status=1/FAILURE)
Sep 13 08:59:00 ip-172-31-46-255 systemd[1]: airflow-webserver.service: Main process exited, code=exited, status=1/FAILURE
Sep 13 08:59:00 ip-172-31-46-255 systemd[1]: airflow-webserver.service: Unit entered failed state.
Sep 13 08:59:00 ip-172-31-46-255 systemd[1]: airflow-webserver.service: Failed with result 'exit-code'.
Help is highly appreciated!非常感谢帮助!
The following is airflow-webserver.service
that works for me with a virtual environment:以下是适合我在虚拟环境中使用的
airflow-webserver.service
:
[Unit]
Description=Airflow webserver daemon
After=network.target postgresql.service mysql.service redis.service rabbitmq-server.service
Wants=postgresql.service mysql.service redis.service rabbitmq-server.service
[Service]
PIDFile=/run/airflow/webserver.pid
EnvironmentFile=/etc/default/airflow
User=airflow
Group=airflow
Type=simple
ExecStart=/usr/bin/bash -c 'source /usr/local/airflow/venv/bin/activate ; airflow webserver --pid /run/airflow/webserver.pid'
Restart=on-failure
RestartSec=5s
PrivateTmp=true
[Install]
WantedBy=multi-user.target
One completetely different, but much easier solution which worked for me:一个完全不同但更简单的解决方案对我有用:
airflow
airflow
apache-airflow
in environment as airflow
userairflow
用户身份在环境中安装apache-airflow
airflow webserver --port 8080 -D
airflow webserver --port 8080 -D
This runs airflow as daemon background process.这将气流作为守护进程后台进程运行。 You can just switch back to other user and continue creating dags, etc.
您可以切换回其他用户并继续创建 dag 等。
Answer given by @kaxil kinda worked for me! @kaxil 给出的答案对我有用! But with slight modifications.
但稍加修改。
I have my Airflow installed(Using Anaconda)and is at the location:我安装了气流(使用 Anaconda)并且位于以下位置:
#-> whereis airflow
airflow: /root/anaconda3/bin/airflow
Adding the below line in airflow-webserver.service
file solved the issue for me.在
airflow-webserver.service
文件中添加以下行为我解决了这个问题。
ExecStart=/usr/bin/bash -c 'source /root/anaconda3/bin/activate ; airflow webserver --pid /run/airflow/webserver.pid'
I got another error after it saying, Error: /run/airflow doesn't exist. Can't create pidfile.
在说
Error: /run/airflow doesn't exist. Can't create pidfile.
之后,我又收到了另一个错误Error: /run/airflow doesn't exist. Can't create pidfile.
Error: /run/airflow doesn't exist. Can't create pidfile.
, Adding the below under [Service]
solved that issue as well , 在
[Service]
下添加以下内容也解决了该问题
RuntimeDirectory=airflow
RuntimeDirectoryMode=0775
This solution worked for me这个解决方案对我有用
Create directory to keep airflow pid files创建目录以保存气流 pid 文件
sudo mkdir /run/airflow sudo chown -R airflow:airflow /run/airflow/
My Airflow package is installed in Python3 virtual environment , virtual environment is created in this directory /opt/python3_6-virtual_envs/airflow_env/
, so this path you may need to be modified according to your python3 virtual environment path我的Airflow包安装在Python3虚拟环境中,虚拟环境创建在这个目录
/opt/python3_6-virtual_envs/airflow_env/
,所以这个路径你可能需要根据你的python3虚拟环境路径修改
Webserver systemd file网络服务器系统文件
[Unit]
Description=Airflow Webserver daemon
[Service]
User=airflow
Type=simple
ExecStart= /opt/python3_6-virtual_envs/airflow_env/bin/python /opt/python3_6-virtual_envs/airflow_env/bin/airflow webserver -p 8080 --pid /run/airflow/webserver.pid
Environment="PATH=/opt/python3_6-virtual_envs/airflow_env/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/airflow/.local/bin:/home/airflow/bin"
[Install]
WantedBy=multi-user.target
[Unit]
Description=Airflow Scheduler daemon
[Service]
User=airflow
Type=simple
ExecStart= /opt/python3_6-virtual_envs/airflow_env/bin/python /opt/python3_6-virtual_envs/airflow_env/bin/airflow scheduler --pid /run/airflow/scheduler.pid
Environment="PATH=/opt/python3_6-virtual_envs/airflow_env/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/airflow/.local/bin:/home/airflow/bin"
[Install]
WantedBy=multi-user.target
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.