[英]Startup script doesn't run Airflow webserver VM GCP
I'm trying to automatically run airflow webserver and scheduler in a VM upon boot using startup scripts just followed the documentation here: https://cloud.google.com/compute/docs/instances/startup-scripts/linux .我正在尝试在启动时使用启动脚本在 VM 中自动运行 airflow 网络服务器和调度程序,只需遵循此处的文档: https://cloud.google.com/compute/docs/instances/startup-scripts/linux 。 Here is my script:
这是我的脚本:
export AIRFLOW_HOME=/home/name/airflow
cd /home/name/airflow
nohup airflow scheduler >> scheduler.log &
nohup airflow webserver -p 8080 >> webserver.log &
The.log files are created which means the script is been executed but the webserver and the scheduler don't.创建了 .log 文件,这意味着脚本已执行,但网络服务器和调度程序未执行。
Any apparent reason?有什么明显的原因吗?
I have tried replicating the Airflow webserver Startup script on GCP VM using the document .我尝试使用文档在 GCP VM 上复制 Airflow 网络服务器启动脚本。
Steps followed to run Airflow webserver Startup script on GCP VM:在 GCP VM 上运行 Airflow 网络服务器启动脚本的步骤如下:
Create a Service Account.创建服务帐户。 Give minimum access to BigQuery with the role of BigQuery Job User and Dataflow with the role of Dataflow Worker.
以 BigQuery Job User 角色授予对 BigQuery 的最低访问权限,以 Dataflow Worker 角色授予对 Dataflow 的最低访问权限。 Click Add Key/Create new key/Done.
单击添加密钥/创建新密钥/完成。 This will download a JSON file.
这将下载一个 JSON 文件。
Create a Compute Engine instance.创建计算引擎实例。 Select the Service Account created.
Select 创建的服务帐户。
Install Airflow libraries.安装 Airflow 个库。 Create a virtual environment using miniconda.
使用 miniconda 创建虚拟环境。
Init your metadata database and register at least one admin user using command:初始化您的元数据数据库并使用命令注册至少一个管理员用户:
airflow db init airflow users create -r Admin -u username -p mypassword -e example@mail.com -f yourname -l lastname
Whitelist IP for port 8080. Create Firewall Rule and add firewall rule on GCP VM instance.端口 8080 的白名单 IP。创建防火墙规则并在 GCP VM 实例上添加防火墙规则。 Now go to terminal and start web server using command airflow webserver -p 8080 .
现在 go 到终端并使用命令airflow webserver -p 8080启动 web 服务器。
Open another terminal and start the Scheduler.打开另一个终端并启动调度程序。
export AIRFLOW_HOME=/home/acachuan/airflow-medium cd airflow-medium conda activate airflow-medium airflow db init airflow scheduler
We want our Airflow to start immediately after the Compute Engine starts.我们希望我们的 Airflow 在 Compute Engine 启动后立即启动。 So we can create a Cloud Storage bucket and then create a script, upload the file and keep it as a backup.
所以我们可以创建一个 Cloud Storage 存储桶,然后创建一个脚本,上传文件并保留它作为备份。
Now pass a Linux startup script from Cloud Storage to a VM.现在将 Linux 启动脚本从 Cloud Storage 传递到 VM。 Refer Passing a startup script that is stored in Cloud Storage to an existing VM .
请参阅将存储在 Cloud Storage 中的启动脚本传递给现有 VM 。 You can also pass a startup script to an existing VM.
您还可以将启动脚本传递给现有 VM。
Note: PermissionDenied desc = The caller does not have permission means you don't have sufficient permissions, you need to request access from your project, folder, or organization admin.注意: PermissionDenied desc = The caller does not have permission 表示您没有足够的权限,您需要向您的项目、文件夹或组织管理员申请访问权限。 Depending on the assets you are trying to export.
取决于您尝试导出的资产。 And to access files which are created by root users you need read, write or execute permissions.
要访问由 root 用户创建的文件,您需要读取、写入或执行权限。 Refer File permissions .
请参阅文件权限。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.