将 EC2 实例作为工作程序添加到 Amazon MWAA

Question

I am currently using Amazon MWAA as my Airflow. I want to have 2 types of workers nodes but currently MWAA doesn't support it. I want to have:

1. High Compute Optimized CPU workers
2. GPU workers

I want to create different queues for both the worker types and submit jobs to these workers nodes.

Is it possible to add an existing EC2 instance (say GPU instance) to MWAA? I only see Start and Stop EC2 operators available.

Does anyone have any pointers on this?

Answer 1

1. If you have an EKS then its possible to define a GPU pool and using Kube.netesPodOperator you can run a docker under a gpu pool.
2. Another solution its ECS (easier to define). you can see a good example to run gpu in Airflow in this article