google cloud composer library installation failure

Question

while attempting to update the installed pypi packages on my composer env, I'm getting the following error -

 UPDATE operation on this environment failed 7 minutes ago with the following error message:
Failed to install pypi packages.
Your service account is missing the required permissions to check the build log for errors. You can access them through Cloud Logging, or by running gcloud builds log fe358f33-51ec-4064-9a75-38af2974976f --project ene-playground.

when reading the build logs, it looks like there's a dependency issue which is clearly isn't permission issue -

google-cloud-vision 1.0.0 has requirement google-api-core[grpc]<2.0.0dev,>=1.14.0, but you have google-api-core 2.7.1. google-cloud-videointelligence 1.16.1 has requirement google-api-core[grpc]<2.0.0dev,>=1.14.0, but you have google-api-core 2.7.1. google-cloud-translate 1.7.0 has requirement google-api-core[grpc]<2.0.0dev,>=1.14.0, but you have google-api-core 2.7.1. google-cloud-texttospeech 1.0.1 has requirement google-api-core[grpc]<2.0.0dev,>=1.14.0, but you have google-api-core 2.7.1. google-cloud-speech 1.3.2 has requirement google-api-core[grpc]<2.0.0dev,>=1.14.0, but you have google-api-core 2.7.1. google-cloud-spanner 1.19.1 has requirement google-api-core[grpc,grpcgcp]<2.0.0dev,>=1.14.0, but you have google-api-core 2.7.1. google-cloud-secret-manager 1.0.0 has requirement google-api-core[grpc]<2.0.0dev,>=1.14.0, but you have google-api-core 2.7.1. google-cloud-memcache 1.0.0 has requirement google-api-core[grpc]<2.0.0dev,>=1.22.2, but you have google-api-core 2.7.1. google-cloud-language 1.3.0 has requirement google-api-core[grpc]<2.0.0dev,>=1.14.0, but you have google-api-core 2.7.1. google-cloud-dlp 1.0.0 has requirement google-api-core[grpc]<2.0.0dev,>=1.14.0, but you have google-api-core 2.7.1. google-cloud-core 1.7.2 has requirement google-api-core<2.0.0dev,>=1.21.0, but you have google-api-core 2.7.1. google-cloud-container 1.0.1 has requirement google-api-core[grpc]<2.0.0dev,>=1.14.0, but you have google-api-core 2.7.1. google-cloud-bigtable 1.7.0 has requirement google-api-core[grpc]<2.0.0dev,>=1.14.0, but you have google-api-core 2.7.1. google-api-python-client 1.12.8 has requirement google-api-core<2dev,>=1.21.0, but you have google-api-core 2.7.1. apache-beam 2.34.0 has requirement avro-python3.=1.9,2.<1.10,0.>=1.8,1. but you have avro-python3 1.10.0. The command '/bin/sh -c bash installer:sh $COMPOSER_PYTHON_VERSION fail' returned a non-zero code: 1 ERROR

before exporting my code to k8sPodOperator I want to understand what's the actual problem but I'm confused between those 2 errors.

PS: I've installed all those libraries on my local Airflow and they work perfectly fine together.

my composer version is: 1.17.7 my airflow version is: 2.1.4

any ideas what's the actual error preventing me from installing the library?

Answer 1

Try installing the library using the latest version of composer and airflow.