[英]AirFlow setup WSL
I am trying to setup AirFlow in WSL and running into problems.我正在尝试在 WSL 中设置 AirFlow 并遇到问题。 I have gone through articles and SO but not able to resolve this.
我已经阅读了文章和 SO,但无法解决这个问题。
I am following the Airlfow tutorial https://airflow.apache.org/docs/stable/tutorial.html to get setup.我正在关注 Airlfow 教程https://airflow.apache.org/docs/stable/tutorial.html进行设置。 I am able to run the run the final script in the tutorial above without errors.
我能够运行上面教程中的最终脚本而不会出错。 Basically this piece of code
基本上这段代码
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from datetime import datetime, timedelta
default_args = {
'owner': 'Airflow',
'depends_on_past': False,
'start_date': datetime(2015, 6, 1),
'email': ['airflow@example.com'],
'email_on_failure': False,
'email_on_retry': False,
'retries': 1,
'retry_delay': timedelta(minutes=5),
# 'queue': 'bash_queue',
# 'pool': 'backfill',
# 'priority_weight': 10,
# 'end_date': datetime(2016, 1, 1),
}
dag = DAG(
'tutorial', default_args=default_args, schedule_interval=timedelta(days=1))
# t1, t2 and t3 are examples of tasks created by instantiating operators
t1 = BashOperator(
task_id='print_date',
bash_command='date',
dag=dag)
t2 = BashOperator(
task_id='sleep',
bash_command='sleep 5',
retries=3,
dag=dag)
templated_command = """
{% for i in range(5) %}
echo "{{ ds }}"
echo "{{ macros.ds_add(ds, 7)}}"
echo "{{ params.my_param }}"
{% endfor %}
"""
t3 = BashOperator(
task_id='templated',
bash_command=templated_command,
params={'my_param': 'Parameter I passed in'},
dag=dag)
t2.set_upstream(t1)
t3.set_upstream(t1)
I am then able to execute it without errors using然后我可以使用它来执行它而不会出错
python ~/airflow/dags/FirstTest.py
However when I try to run the command但是,当我尝试运行命令时
airflow list_dags
I get an error saying我收到一条错误消息
airflow: command not found
I was able to do a pip install apache-airflow and therefore the original script could run.我能够执行 pip install apache-airflow ,因此可以运行原始脚本。
However, according to this SO forum question (though a different platform) Getting bash: airflow: command not found但是,根据这个 SO 论坛问题(虽然是不同的平台) Getting bash:airflow: command not found
a SUDO pip install pip fixed it, but in my case it gives me an error saying一个 SUDO pip install pip 修复了它,但在我的情况下它给了我一个错误说
'/home/sum/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled
I have then used the -H flag meaning然后我使用了 -H 标志的含义
sudo -H pip3 install apache-airflow
But it seemed it was not hitting fiddler (127.0.0.1:8888) which I have as proxy as thus got a connection refused error但它似乎没有击中我作为代理的小提琴手(127.0.0.1:8888),因此出现连接拒绝错误
Therefore I modified it to include proxy explicitly using因此,我将其修改为显式包含代理
sudo pip3 --proxy=http://127.0.0.1:8888 install apache-airflow
But then I ran into the issue that all the issues that it went to pypi and not npm.sys.dom which I wanted.但是后来我遇到了一个问题,它所有的问题都交给了 pypi 而不是我想要的 npm.sys.dom。
In my /etc/profile file I have在我的 /etc/profile 文件中,我有
export http_proxy=http://127.0.0.1:8888
export https_proxy=http://127.0.0.1:8888
export pip_index_url=http://npm.sys.dom:81/pypi/Python/simple
export pip_trusted_host=npm.sys.dom
But somehow it seems VS code does not pick it up.但不知何故,似乎 VS 代码并没有把它捡起来。
I tried setting the ".profile" file within VS code which came up after I chose to open the folder corresponding to my home directory, using the same 4 lines as above.我尝试在 VS 代码中设置“.profile”文件,该文件是在我选择打开与我的主目录对应的文件夹后出现的,使用与上面相同的 4 行。
But still the sudo pip install would not pick it up .但仍然 sudo pip install 不会拿起它。
Finally I got around by doing最后我通过做
sudo pip3 --proxy=http://127.0.0.1:8888 trusted-host npm.sys.dom index-url http://npm.sys.dom:81/pypi/Python/simple install apache-airflow
But now when I run但是现在当我跑
airflow list_dags
I get the dags that I have but I also get a bunch of errors before getting the output我得到了我拥有的 dags,但在获得输出之前我也遇到了一堆错误
ERROR - Failed on pre-execution callback using <function default_action_log at 0x7fa84c2db510>
Traceback (most recent call last):
File "/home/sum/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1246, in _execute_context
cursor, statement, parameters, context
File "/home/sum/.local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 581, in do_execute
cursor.execute(statement, parameters)
sqlite3.OperationalError: no such table: log
......
(Background on this error at: http://sqlalche.me/e/e3q8)
When I look at http://sqlalche.me/e/e3q8 , it shows something related to connection not being released which I will investigate further.当我查看http://sqlalche.me/e/e3q8 时,它显示了与未释放连接相关的内容,我将进一步调查。
My question is: All the workarounds that I had to do for installing airflow , is there a cleaner way and what did I miss?我的问题是:我为安装气流而必须采取的所有解决方法,是否有更清洁的方法,我错过了什么? Just, so that I do not have keep doing this same stuff again and again.
只是,这样我就不会一次又一次地做同样的事情。
PS: Not super well versed in Linux. PS:不是非常精通Linux。 Using Ubuntu 18.04 for Windows.
在 Windows 上使用 Ubuntu 18.04。
Finally got around this one by:终于通过以下方式解决了这个问题:
echo 'export HTTP_PROXY="127.0.0.1:8888"' >> ~/.bashrc
echo 'export HTTPS_PROXY="127.0.0.1:8888"' >> ~/.bashrc
apt-get install python3-venv
#Please replace or choose your own path. The one below is a sample
python3 -m venv /home/usrname/airflow
source /home/usrname/airflow/bin/activate
#finally install airflow. Remember to install setuptools before. I also had other packages to install
pip install setuptools_git "pymssql<3.0" pyodbc
pip install apache-airflow
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.