How to trigger Google Composer Airflow dag using appscript?

Question

I want to trigger a Google Composer airflow dag using Appscript. Is there any way to do it via rest API or another way.

If it is possible please suggest the solution.

Answer 1

Airflow has an endpoint that allows to trigger a DAG through its REST API , however it's not possible to access it directly, since within the Cloud Composer architecture , the Airflow web server is located under an App Engine flexible environment. By default, the Airflow web server is integrated with Identity-Aware Proxy (IAP) and authentication is required.

Based on that, I found an example in the Cloud Composer documentation, that guides you to trigger a DAG using Cloud Functions , although the code is in JavaScript I don't think it's possible to execute it by Google App Script.

On the other hand, a workaround is to follow the Triggering DAGs guide changing some settings as follows.

1. In the creation of the function instead of setting the trigger type as Cloud Storage set it as HTTP, and check the “Allow unauthenticated invocations” for test purpose. An URL will be displayed, the goal is that every time that URL is accessed the DAG is executed.

2. Modify the first part of the index.js file, since no data would be passed as parameters and also the makeIapPostRequest function to return the response of the API call.

    exports.triggerDag = async (req, res) => { // Modification // Fill in your Composer environment information here. // The project that holds your function const PROJECT_ID = 'your-project-id'; // Navigate to your webserver's login page and get this from the URL const CLIENT_ID = 'your-iap-client-id'; // This should be part of your webserver's URL: // {tenant-project-id}.appspot.com const WEBSERVER_ID = 'your-tenant-project-id'; // The name of the DAG you wish to trigger const DAG_NAME = 'composer_sample_trigger_response_dag'; // Other constants const WEBSERVER_URL = `https://${WEBSERVER_ID}.appspot.com/api/experimental/dags/${DAG_NAME}/dag_runs`; const USER_AGENT = 'gcf-event-trigger'; const BODY = {conf: ''}; // Modification // Make the request try { const iap = await authorizeIap(CLIENT_ID, PROJECT_ID, USER_AGENT); const apiReponse = await makeIapPostRequest(WEBSERVER_URL, BODY, iap.idToken, USER_AGENT); // Modification res.status(200).send('DAG_running!'); // Modification } catch (err) { console.error('Error authorizing IAP:', err.message); throw new Error(err); } }; const makeIapPostRequest = async (url, body, idToken, userAgent) => { const res = await fetch(url, { method: 'POST', headers: { 'User-Agent': userAgent, Authorization: `Bearer ${idToken}`, }, body: JSON.stringify(body), }); if (!res.ok) { const err = await res.text(); console.error('Error making IAP post request:', err.message); throw new Error(err); } return { apiRes: res.ok, // Modification }; };

3. At this point, anything else has to be changed, so in your Script file execute the next instructions in order to trigger the DAG.

    function myFunction() { var response = UrlFetchApp.fetch("Cloud-function-URL"); Logger.log(response.getAllHeaders()); }

4. Finally, verify in the Airflow web interface if the DAG was triggered.