如何通过气流在 s3 中运行 python 脚本

Question

We usually store our code in s3 and call it from airflow to add them as new steps in emr. spark-submit can read data from s3 (like jars and files). But how can i call a python script that is present in s3 from airflow? It is a simple boto3 script. Please provide me some guidance.

Answer 1

I would think that its a round about way of actually running an emr job via ariflow. Why would you even need to add steps.Your easiest option would be to use the ssh operator that connects to emr and then do a spark-submit via the ssh operator. spark-submit will get your code from s3 and and then run the jobs.