[英]Run airflow commands on official Airflow docker-compose
I am using the oficial docker-compose.yml
to run airflow 2.1.4
locally and i'm having trouble understanding what happens when i run some airflow
commands on the airflow-init
service(file below).我正在使用官方的docker-compose.yml
在本地运行airflow 2.1.4
,我无法理解当我在airflow-init
-init 服务上运行一些airflow
命令时会发生什么。
This is the log output: log output这是日志 output:日志 output
I am not understanding why i have repeated instructions.我不明白为什么我有重复的指示。 Should i be running this command
from airflow-init
in a separate entrypoint script?我应该在单独的入口airflow-init
运行这个command
吗?
Thanks谢谢
airflow-init:
<<: *airflow-common
entrypoint: /bin/bash
# yamllint disable rule:line-length
command:
- -c
- |
function ver() {
printf "%04d%04d%04d%04d" $${1//./ }
}
airflow_version=$$(gosu airflow airflow version)
airflow_version_comparable=$$(ver $${airflow_version})
min_airflow_version=2.1.0
min_airflow_version_comparable=$$(ver $${min_airflow_version})
if (( airflow_version_comparable < min_airflow_version_comparable )); then
echo
echo -e "\033[1;31mERROR!!!: Too old Airflow version $${airflow_version}!\e[0m"
echo "The minimum Airflow version supported: $${min_airflow_version}. Only use this or higher!"
echo
exit 1
fi
if [[ -z "${AIRFLOW_UID}" ]]; then
echo
echo -e "\033[1;33mWARNING!!!: AIRFLOW_UID not set!\e[0m"
echo "If you are on Linux, you SHOULD follow the instructions below to set "
echo "AIRFLOW_UID and AIRFLOW_GID environment variables, otherwise files will be owned by root."
echo "For other operating systems you can get rid of the warning with manually created .env file:"
echo " See: https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html#setting-the-right-airflow-user"
echo
fi
one_meg=1048576
mem_available=$$(($$(getconf _PHYS_PAGES) * $$(getconf PAGE_SIZE) / one_meg))
cpus_available=$$(grep -cE 'cpu[0-9]+' /proc/stat)
disk_available=$$(df / | tail -1 | awk '{print $$4}')
warning_resources="false"
if (( mem_available < 4000 )) ; then
echo
echo -e "\033[1;33mWARNING!!!: Not enough memory available for Docker.\e[0m"
echo "At least 4GB of memory required. You have $$(numfmt --to iec $$((mem_available * one_meg)))"
echo
warning_resources="true"
fi
if (( cpus_available < 2 )); then
echo
echo -e "\033[1;33mWARNING!!!: Not enough CPUS available for Docker.\e[0m"
echo "At least 2 CPUs recommended. You have $${cpus_available}"
echo
warning_resources="true"
fi
if (( disk_available < one_meg * 10 )); then
echo
echo -e "\033[1;33mWARNING!!!: Not enough Disk space available for Docker.\e[0m"
echo "At least 10 GBs recommended. You have $$(numfmt --to iec $$((disk_available * 1024 )))"
echo
warning_resources="true"
fi
if [[ $${warning_resources} == "true" ]]; then
echo
echo -e "\033[1;33mWARNING!!!: You have not enough resources to run Airflow (see above)!\e[0m"
echo "Please follow the instructions to increase amount of resources available:"
echo " https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html#before-you-begin"
echo
fi
mkdir -p /sources/logs /sources/dags /sources/plugins
chown -R "${AIRFLOW_UID}:${AIRFLOW_GID}" /sources/{logs,dags,plugins}
/entrypoint airflow connections import /sources/connections.json
/entrypoint airflow variables import /sources/variables.json
# yamllint enable rule:line-length
environment:
<<: *airflow-common-env
_AIRFLOW_DB_UPGRADE: 'true'
_AIRFLOW_WWW_USER_CREATE: 'true'
_AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}
_AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}
user: "0:${AIRFLOW_GID:-0}"
volumes:
- .:/sources
Because you are using _PIP_ADDITIONAL_REQUIREMENTS which is just a convenience stuff for very quickly installing dependencies if you want to try things out.因为您使用的是 _PIP_ADDITIONAL_REQUIREMENTS,如果您想尝试一下,这只是一个方便的东西,可以快速安装依赖项。
You should NOT use it for anything else precisely because you are going to observe what you see here - that installing new dependencies dynamically happens EVERY TIME you run ANY command.您不应该将它用于其他任何事情,因为您将观察您在此处看到的内容 - 每次运行任何命令时都会动态安装新的依赖项。
As explained here: https://airflow.apache.org/docs/docker-stack/entrypoint.html#installing-additional-requirements you should build and use your own custom image when you want to add dependencies.如此处所述: https://airflow.apache.org/docs/docker-stack/entrypoint.html#installing-additional-requirements当您想要添加依赖项时,您应该构建并使用您自己的自定义映像。 Read more here: https://airflow.apache.org/docs/docker-stack/build.html where you find instructions and plenty of useful examples.在此处阅读更多信息: https://airflow.apache.org/docs/docker-stack/build.html在这里您可以找到说明和大量有用的示例。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.