Google Cloud Composer 单调度程序与多调度程序

Question

Background: I have a composer environment which has 40 dags, out of which there are few dags which are making API calls for downloading the files and there are some which are moving files from gcs bucket to gcs bucket and some dags move files from gcs bucket to BigQuery.

Scheduler configs:

Number of Schedulers:1, CPU: 1 vCPUs, Memory: 1.75 GB, Storage: 1 GB

It used all the 100% of cpu and dag parsig time was more then a 60 seconds

According to the documentation CPU should not exceed more then 80% and DAG parsing time should not be more then 10 seconds. So I did some test with the configs.

Test 1: Added one more scheduler with same 1 vCPUs, 1 GB memory and 1GB storage, now because of two scheduler we have 2 vCPUs, 2 GB memory and 2 GB of storage For some reason it sill occupied all the 100% of scheduler cpu and dag parsing time was fell to 10-20 seconds.

Test 2: With the 1 scheduler I increased cpu to 1.75 vCPUs, memory was 1.75 GB and storage to 2 GB. For some reason it used the average cpu to 1.2 less then 80% of usage in cpu and DAG parsing time fell to less then 4 seconds.

I am not able to understand the actual reason isn't having two schedulers should be faster? is there something which I am missing?

Answer 1

Airflow scheduler is used to monitor the tasks and DAGs and trigger the task instance once their dependencies are complete. Multiple schedulers can be added to distribute load but that does not improve the performance of the Airflow. One schedule might give better performance than multiple schedulers which may happen due to underutilization of the scheduler which results in consumption of resources but not improving performance or contributing to performance.

The performance of the scheduler depends on the number of Airflow workers, the number of DAGs and tasks that run in your environment, and the configuration of both Airflow and the environment. It's recommended to start with two schedulers and monitor the performance and scale according to your requirement. You can check this documentation for more information.