google-cloud dependencies getting overriden in python app engine

Question

We are using GAE standard and as per google documentation, we are expected to vendor dependencies in some folder using pip install requirement-name -t lib/ .

We want to use both BigQuery & pub-sub in the same GAE project which is I think not a very rare combination.

Now the issue is at a time we are able to install only one of the client libraries as both share google>cloud namespace whichever is installed later will override earlier dependency.

Screenshots:-

1) When only BigQuery client library is installed :-

2) After installing pip install -t lib/ google-cloud-pubsub

What can be the issue and solution for the same?

UPDATE :-

Further tried installing these two dependencies in the virtual environment without vendoring in lib/ folder, there both dependencies are getting installed without overwriting each other. So looked like the issue is with vendoring in folder.

Screenshot for the same without vendoring dependencies.

Answer 1

You should install all the dependencies in a single command:

$ pip install -t lib/ google-cloud-bigquery google-cloud-pubsub

This will install both packages along side each other:

$ ls lib/google/cloud
__pycache__  _testing.py  environment_vars.py  obsolete.py   pubsub_v1
_helpers.py  bigquery     exceptions.py        operation.py
_http.py     client.py    iam.py               pubsub.py

Installing the dependencies from a requirements file should have the same effect:

$ cat requirements.txt
google-cloud-bigquery
google-cloud-pubsub
$ pip install -t lib/ -r requirements.txt

Make sure to start with an empty lib/ folder, and with the latest version of pip :

pip install -U pip