I am looking for a process to pull Marketo data using Airflow. It seems that Airflow has no native Marketo webhook, did I understand that correctly ?
Or is there a way to use an existing Airflow operator/webhook to pull data from Marketo? Thanks
Most Operators for 3rd parties just wrap around APIs, Marketo has a REST api . You could use a PythonOperator
with requests
pull your data.
An example DAG could be:
import pendulum
from airflow.decorators import dag, task
from airflow.providers.amazon.aws.operators.s3 import S3CreateObjectOperator
@dag(
schedule_interval=None,
start_date=pendulum.yesterday(tz="Europe/London"),
catchup=False,
)
def marketo_api_dag():
@task.virtualenv(system_site_packages=False, requirements=["requests"])
def pull_from_marketo():
import requests
import json
response = requests.get() # pull from marketo API using requests
return json.dumps({}) # response formatted how you need it
marketo_api_response_data = pull_from_marketo()
save_marketo_to_s3 = S3CreateObjectOperator(
task_id="save_marketo_to_s3", s3_bucket="s3bucket", s3_key="/path/to/save", data=marketo_api_response_data
)
marketo_api_response_data >> save_marketo_to_s3
marketo_api_dag_run = marketo_api_dag()
apache-airflow-providers-amazon
.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.