简体   繁体   中英

Composer/Airflow - Adding a dependency while running a DAG/pipeline?

this might be silly for many, but I did some research and couldn't find the answer, so that's why I'm asking:

  • When you're adding a new dependency, let's say Pandas, to you Composer Machine in GCP, does it mess up with the running of your pipelines? Because the 'server' kind of restarts and keep thinking for at least 5 mins when adding a new dependency. Again, my question is: can this process of adding a dependency makes your DAGs fail or your pipelines crash or something like that? Thanks!

I believe this could mess up with your running DAGs.
When you add a new dependency, the environment must upgrade and you need to pause all DAGs and wait for in-progress tasks to finish before upgrading ( Reference ).

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM