How we can execute Bash script (which present in Compute engine VM) from GCP composer (DAG)?

Question

I have one sh file which is present in GCP compute engine VM instance. In the same service account or same project I want to execute that bash script from GCP composer DAG. How we can execute?

I found one solution I am not sure it's correct. There is a way using "SSHExecuteOperator"; then we need to create SSH connection. Do we still need the KeyFile, when composer and VM has the shell scripts running with same service account?

Answer 1

You can do something like this, the ssh_conn_id will be setup in the Composer UI

from datetime import timedelta, datetime
import airflow
from airflow import DAG
from airflow.contrib.operators.ssh_operator import SSHOperator

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'email': ['airflow@example.com'],
    'email_on_failure': False,
    'email_on_retry': False,
    'start_date': datetime.now() - timedelta(minutes=20),
    'retries': 1,
    'retry_delay': timedelta(minutes=5),
}

dag = DAG(dag_id='testing_stuff',
          default_args=default_args,
          schedule_interval='0,10,20,30,40,50 * * * *',
          dagrun_timeout=timedelta(seconds=120))

t1_bash = """
echo 'Hello World'
"""

t1 = SSHOperator(
    ssh_conn_id=<<Connection Id in Composer UI>>,
    task_id='test_ssh_operator',
    command=t1_bash,
    dag=dag)