Startup script doesn't run Airflow webserver VM GCP

Question

I'm trying to automatically run airflow webserver and scheduler in a VM upon boot using startup scripts just followed the documentation here: https://cloud.google.com/compute/docs/instances/startup-scripts/linux . Here is my script:

export AIRFLOW_HOME=/home/name/airflow
cd /home/name/airflow
nohup airflow scheduler >> scheduler.log &
nohup airflow webserver -p 8080 >> webserver.log &

The.log files are created which means the script is been executed but the webserver and the scheduler don't.

Any apparent reason?

Answer 1

I have tried replicating the Airflow webserver Startup script on GCP VM using the document .

Steps followed to run Airflow webserver Startup script on GCP VM:

1. Create a Service Account. Give minimum access to BigQuery with the role of BigQuery Job User and Dataflow with the role of Dataflow Worker. Click Add Key/Create new key/Done. This will download a JSON file.

2. Create a Compute Engine instance. Select the Service Account created.

3. Install Airflow libraries. Create a virtual environment using miniconda.

4. Init your metadata database and register at least one admin user using command:

    airflow db init airflow users create -r Admin -u username -p mypassword -e example@mail.com -f yourname -l lastname

5. Whitelist IP for port 8080. Create Firewall Rule and add firewall rule on GCP VM instance. Now go to terminal and start web server using command airflow webserver -p 8080 .

6. Open another terminal and start the Scheduler.

    export AIRFLOW_HOME=/home/acachuan/airflow-medium cd airflow-medium conda activate airflow-medium airflow db init airflow scheduler

7. We want our Airflow to start immediately after the Compute Engine starts. So we can create a Cloud Storage bucket and then create a script, upload the file and keep it as a backup.

8. Now pass a Linux startup script from Cloud Storage to a VM. Refer Passing a startup script that is stored in Cloud Storage to an existing VM . You can also pass a startup script to an existing VM.

Note: PermissionDenied desc = The caller does not have permission means you don't have sufficient permissions, you need to request access from your project, folder, or organization admin. Depending on the assets you are trying to export. And to access files which are created by root users you need read, write or execute permissions. Refer File permissions .