Airflow execute python script through bash operator

Question

I have a python script test2.py to connect to a remote server and execute the command. as below. This works on the command line.

Passing parameters as JSON and getting the response in JSON this works when executed as below in the command line.

python3.6 test2.py  '{"hostname": "<server>", "username":"<test>", "password":"<test1>", "command1":"hostname"}'

I am trying to execute same through the airflow

from __future__ import print_function
from airflow.operators import BashOperator
from airflow.models import DAG
from datetime import datetime, timedelta


default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime(2018, 9, 1),
    'email_on_failure': False,
    'email_on_retry': False,
    'schedule_interval': '@daily',
    'retries': 1,
    'retry_delay': timedelta(seconds=5),
}

dag = DAG(
    dag_id='DAG-3',
    default_args=default_args,
    dagrun_timeout=timedelta(minutes=10)
    )

cmd_command = "python3.6 /root/test2.py '{{"hostname": "<server>", "username":"<test>", "password":"<test1>", "command1":"hostname"}}'"


t = BashOperator(
     task_id = 'some_id',
     bash_command = cmd_command,
     dag = dag)
 

I am seeing below error related to syntax.?

cmd_command = "python3.6 /root/test2.py '{{"hostname": "<server>", "username":"<test>", "password":"<test1>", "command1":"hostname"}}'"
                                                       ^
SyntaxError: invalid syntax

Can you please help

Thank you

Answer 1

You use double quotes for JSON, but Python interprets them as start or end of a string. One way to resolve this is to escape double quotes inside JSON:

cmd_command = "python3.6 /root/test2.py '{\"hostname\": \"<server>\", \"username\":\"<test>\", \"password\":\"<test1>\", \"command1\":\"hostname\"}'"