__init__.py 中的 google.cloud 命名空间导入错误

Question

I have read through at least a dozen different stackoverflow questions that all present the same basic problem and have the same basic answer: either the module isn't installed correctly or the OP is doing the import wrong.

In this case, I am trying to do from google.cloud import secretmanager_v1beta1 .

It works in my airflow container when I run airflow dags or if I run pytest tests/dags/test_my_dag.py . However, if I run cd dags; python -m my_dag cd dags; python -m my_dag or cd dags; python my_dag.py cd dags; python my_dag.py I get this error:

from google.cloud import secretmanager as secretmanager
ImportError: cannot import name 'secretmanager' from 'google.cloud' (unknown location)

I can add from google.cloud import bigquery in the line right above this line and that works OK. It appears to literally just be a problem with this particular package.

Why does it matter if pytest and airflow commands succeed? Because, I have another environment where I am trying to run dataflow jobs from the command-line and I get this same error. And unfortunately I don't think I can bypass this error in that environment for several reasons.

UPDATE 6

I have narrowed down the error to an issue with the google.cloud namespace and the secretmanager package within that namespace in the __init__.py file.

If I add from google.cloud import secretmanager to airflow/dags/__init__.py and then try to run python -m dags.my_dag.py , I receive this error but with a slightly different stacktrace:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/runpy.py", line 183, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/local/lib/python3.7/runpy.py", line 109, in _get_module_details
    __import__(pkg_name)
  File "/workspace/airflow/dags/__init__.py", line 3, in <module>
    from google.cloud import secretmanager
ImportError: cannot import name 'secretmanager' from 'google.cloud' (unknown location)

OLD INFORMATION

I am 95% sure that it's still a path problem and that pytest and airflow are fixing something I'm not aware of that isn't handled when I try to manually run the python script.

Things I have tried:

cd /airflow; python setup.py develop --user
cd /airflow; pip install -e . --user
cd /airflow/dags; pip install -r requirements.txt --user

UPDATE

As per requests in the comments, here are the contents of requirements.txt :

boto3>=1.7.84
google-auth==1.11.2
google-cloud-bigtable==1.2.1
google-cloud-bigquery==1.24.0
google-cloud-spanner==1.14.0
google-cloud-storage==1.26.0
google-cloud-logging==1.14.0
google-cloud-secret-manager>=0.2.0
pycloudsqlproxy>=0.0.15
pyconfighelper>=0.0.7
pymysql==0.9.3
setuptools==45.2.0
six==1.14.0

And I accidentally omitted the --user flags from the pip and python installation command examples above. In my container environment everything is installed into the user's home directory using --user and NOT in the global site-packages directory.

UPDATE 2

I've added the following code to the file that is generating the error:

print('***********************************************************************************')
import sys
print(sys.path)
from google.cloud import secretmanager_v1beta1 as secretmanager
print('secretmanager.__file__: {}'.format(secretmanager.__file__))

From airflow list_dags :

['/home/app/.local/bin', '/usr/local/lib/python37.zip', '/usr/local/lib/python3.7', '/usr/local/lib/python3.7/lib-dynload', '/home/app/.local/lib/python3.7/site-packages', '/home/app/.local/lib/python3.7/site-packages/Jeeves-0.0.1-py3.7.egg', '/home/app/.local/lib/python3.7/site-packages/google_cloud_secret_manager-0.2.0-py3.7.egg', '/home/app/.local/lib/python3.7/site-packages/pyconfighelper-0.0.7-py3.7.egg', '/home/app/.local/lib/python3.7/site-packages/click-7.1.1-py3.7.egg', '/workspace/airflow', '/usr/local/lib/python3.7/site-packages', '/workspace/airflow/dags', '/workspace/airflow/config', '/workspace/airflow/plugins']
secretmanager.__file__: /home/app/.local/lib/python3.7/site-packages/google_cloud_secret_manager-0.2.0-py3.7.egg/google/cloud/secretmanager_v1beta1/__init__.py

From python my_dag.py :

['/workspace/airflow/dags', '/usr/local/lib/python37.zip', '/usr/local/lib/python3.7', '/usr/local/lib/python3.7/lib-dynload', '/home/app/.local/lib/python3.7/site-packages', '/home/app/.local/lib/python3.7/site-packages/Jeeves-0.0.1-py3.7.egg', '/home/app/.local/lib/python3.7/site-packages/google_cloud_secret_manager-0.2.0-py3.7.egg', '/home/app/.local/lib/python3.7/site-packages/pyconfighelper-0.0.7-py3.7.egg', '/home/app/.local/lib/python3.7/site-packages/click-7.1.1-py3.7.egg', '/home/app/.local/lib/python3.7/site-packages/icentris_ml_airflow-0.0.0-py3.7.egg', '/usr/local/lib/python3.7/site-packages']

UPDATE 3 tree airflow/dags

airflow/dags
├── __init__.py
├── __pycache__
│   ├── __init__.cpython-37.pyc
│   ├── bq_to_cs.cpython-37.pyc
│   ├── bq_to_wrench.cpython-37.pyc
│   ├── fetch_cloudsql_tables-bluesun.cpython-37.pyc
│   ├── fetch_cloudsql_tables.cpython-37.pyc
│   ├── fetch_app_tables-bluesun.cpython-37.pyc
│   ├── fetch_app_tables.cpython-37.pyc
│   ├── gcs_to_cloudsql.cpython-37.pyc
│   ├── gcs_to_s3.cpython-37.pyc
│   ├── lake_to_staging.cpython-37.pyc
│   ├── schedule_dfs_sql_to_bq-bluesun.cpython-37.pyc
│   ├── schedule_dfs_sql_to_bq.cpython-37.pyc
│   ├── app_to_bq_initial_load-bluesun.cpython-37.pyc
│   ├── app_to_lake-bluesun.cpython-37.pyc
│   └── app_to_lake.cpython-37.pyc
├── bq_to_wrench.py
├── composer_variables.json
├── my_ml_airflow.egg-info
│   ├── PKG-INFO
│   ├── SOURCES.txt
│   ├── dependency_links.txt
│   └── top_level.txt
├── lake_to_staging.py
├── libs
│   ├── __init__.py
│   ├── __pycache__
│   │   ├── __init__.cpython-37.pyc
│   │   ├── checkpoint.cpython-37.pyc
│   │   └── utils.cpython-37.pyc
│   ├── checkpoint.py
│   ├── io
│   │   ├── __init__.py
│   │   ├── __pycache__
│   │   │   └── __init__.cpython-37.pyc
│   │   └── gcp
│   │       ├── __init__.py
│   │       ├── __pycache__
│   │       │   ├── __init__.cpython-37.pyc
│   │       │   └── storage.cpython-37.pyc
│   │       └── storage.py
│   ├── shared -> /workspace/shared/
│   └── utils.py
├── requirements.txt
├── table_lists
│   └── table-list.json
└── templates
    └── sql
        ├── lake_to_staging.contacts.sql
        ├── lake_to_staging.orders.sql
        └── lake_to_staging.users.sql

11 directories, 41 files

UPDATE 4

I tried fixing it so that sys.path looked the same when running python dags/my_dag.py as it does when running airflow list_dags or pytest test_my_dag.py .

Still get the same error.

Looking at a more recent version of documentation, I noticed that you should be able to just do from google.cloud import secretmanager . Which gave me the same result (works with airflow and pytest, not when trying to run directly).

At this point, my best guess is that it has something to do with namespace magic, but I'm not sure?

Answer 1

它必须通过终端pip install google-cloud-secret-manager ： pip install google-cloud-secret-manager因为包名不是 secretmanager 而是 google-cloud-secret-manager

Answer 2

After much trial and error, the issue is that currently one cannot import secretmanager from the google.cloud namespace if one has not previously imported another package from google.cloud.

Ex.

mod.py

from google.cloud import secretmanager # Fails with error

mod2.py

from google.cloud import bigquery
from google.cloud import secretmanager # Works because the first import initialized the namespace

Answer 3

与诺亚的回答类似，这为我解决了问题，而无需导入不需要的模块：

import google.cloud.secretmanager as secretmanager

Answer 4

I'm using Python 3.8.2 and google-cloud-secret-manager 2.7.1. The following line fixed the issue for me:

from google.cloud import secretmanager_v1beta1 as secretmanager