I installed the google provider with
pip install 'apache-airflow[google]'
and also tried
pip install apache-airflow-providers-google
But I can not find Google Cloud
option when adding a new connection via airflow webserver
, as shown below:
All I found is Google Dataprep
.
I have tried to restart the webserver multiple times but the Google Cloud
option is still not listed. It returns this info when I'm starting the webserver:
{providers_manager.py:215} INFO - Optional provider feature disabled when importing 'airflow.providers.google.leveldb.hooks.leveldb.LevelDBHook' from 'apache-airflow-providers-google' package
Also tried to set lazy_discover_providers = False
on the airflow.cfg
with no luck.
Any help is appreciated. Thanks!
I had a similar problem with MWAA. I had two option to create the connection:
http
(you are not supposed to set all the variables), you can check this doc :{
"extra__google_cloud_platform__project":"<POJECT NAME>",
"extra__google_cloud_platform__key_path":"",
"extra__google_cloud_platform__keyfile_dict":{
"type":"service_account",
"project_id":"<PROJECT ID>",
"private_key_id":"<PRIVATE KEY ID>",
"private_key":"-----BEGIN PRIVATE KEY-----\n<PRIVATE KEY>\n-----END PRIVATE KEY-----\n",
"client_email":"<CLIENT EMAIL>",
"client_id":"<CLIENT ID>",
"auth_uri":"https://<AUTH URI>",
"token_uri":"https://<TOKEN URI>",
"auth_provider_x509_cert_url":"https://<AUTH CERT URI>",
"client_x509_cert_url":"https://<CLIENT CERT URI>"
},
"extra__google_cloud_platform__scope":"",
"extra__google_cloud_platform__num_retries":"10"
}
You can also create it using an environment variable, but it's not secure:
export AIRFLOW_CONN_GOOGLE_CLOUD_DEFAULT='google-cloud-platform://?extra__google_cloud_platform__project=<PROJECT_NAMR>&extra__google_cloud_platform__scope=<SCOPE>&extra__google_cloud_platform__key_path=<KEY_PATH>&extra__google_cloud_platform__num_retries=10'
please try to upgrade your db using 'airflow db upgrade'. Once it did, then you can restart the server. I faced a table creation error while upgrading, then I had to remove the db and initialize it again. Obviously it's not advisable since it removes history. If you don't have anything to preserve, you can check that as well.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.