简体   繁体   中英

"Google Cloud" connection not listed when adding new connection on Airflow Webserver

I installed the google provider with

pip install 'apache-airflow[google]'  

and also tried

pip install apache-airflow-providers-google

But I can not find Google Cloud option when adding a new connection via airflow webserver , as shown below: 在此处输入图像描述

All I found is Google Dataprep .

I have tried to restart the webserver multiple times but the Google Cloud option is still not listed. It returns this info when I'm starting the webserver:

{providers_manager.py:215} INFO - Optional provider feature disabled when importing 'airflow.providers.google.leveldb.hooks.leveldb.LevelDBHook' from 'apache-airflow-providers-google' package

Also tried to set lazy_discover_providers = False on the airflow.cfg with no luck.

Any help is appreciated. Thanks!

I had a similar problem with MWAA. I had two option to create the connection:

  1. using Airflow CLI
  2. from the UI with type http (you are not supposed to set all the variables), you can check this doc :
{
   "extra__google_cloud_platform__project":"<POJECT NAME>",
   "extra__google_cloud_platform__key_path":"",
   "extra__google_cloud_platform__keyfile_dict":{
      "type":"service_account",
      "project_id":"<PROJECT ID>",
      "private_key_id":"<PRIVATE KEY ID>",
      "private_key":"-----BEGIN PRIVATE KEY-----\n<PRIVATE KEY>\n-----END PRIVATE KEY-----\n",
      "client_email":"<CLIENT EMAIL>",
      "client_id":"<CLIENT ID>",
      "auth_uri":"https://<AUTH URI>",
      "token_uri":"https://<TOKEN URI>",
      "auth_provider_x509_cert_url":"https://<AUTH CERT URI>",
      "client_x509_cert_url":"https://<CLIENT CERT URI>"
   },
   "extra__google_cloud_platform__scope":"",
   "extra__google_cloud_platform__num_retries":"10"
}

You can also create it using an environment variable, but it's not secure:

export AIRFLOW_CONN_GOOGLE_CLOUD_DEFAULT='google-cloud-platform://?extra__google_cloud_platform__project=<PROJECT_NAMR>&extra__google_cloud_platform__scope=<SCOPE>&extra__google_cloud_platform__key_path=<KEY_PATH>&extra__google_cloud_platform__num_retries=10'

please try to upgrade your db using 'airflow db upgrade'. Once it did, then you can restart the server. I faced a table creation error while upgrading, then I had to remove the db and initialize it again. Obviously it's not advisable since it removes history. If you don't have anything to preserve, you can check that as well.

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM