尝试建立与Azure Datalake的连接时，Kubernetes Pod中出现“ OAuth2Client：获取令牌请求失败”错误

Question

I have a docker image which creates a connection to Azure Datalake using adlCreds = lib.auth(tenant_id=tenantId, client_secret=application_key, client_id=application_id) in python and access a file. I am executing the docker image in a Kubernetes Pod using Airflow Kubernetes_Pod_Operator. When I instantiate the pod operator, the container in Kubernetes is terminating with the error OAuth2Client:Get Token request failed

I have verified the Azure creds and the connection is working fine when I execute it locally. I am getting this error only when I run it in the cluster using Kubernetes Pod.

        # Creating ADL connection

        def make_azure_enduser():
        logging.info('Creating the Azure Datalake Client...')
        config = dict(azure_tenantid=<tenant_id>'),
                      application_key=<azure_application_key>),
                      application_id=<azure_applicationid>),
                      subscriptionId=<azure_subid>),
                      adlAccountName=<azure_datalake_accountname>)
                      )
        # Format the tenant ID and Account Name as strings to be passed into the object
        application_id = "{}".format(config['application_id'])
        application_key = "{}".format(config['application_key'])
        tenantId = "{}".format(config['azure_tenantid'])
        store_name = "{}".format(config['adlAccountName'])

        # Create the token and the Azure Datalake File System Client
        adlCreds = lib.auth(tenant_id=tenantId, client_secret=application_key, client_id=application_id)

        adlsFileSystemClient = core.AzureDLFileSystem(adlCreds, store_name=store_name)
        return adlsFileSystemClient

        # instantiating Kubernetes_pod_operator in Airflow DAG

        get_adl_file = KubernetesPodOperator(namespace='default',
                                 image="get_adl_file:latest",
                                 image_pull_policy='IfNotPresent',
                                 image_pull_secrets='acr-registry',
                                 # cmds=["/bin/sh", "-c", "sleep 500"],
                                 # arguments=["sleep 500"],
                                 labels={"foo": "bar"},
                                 name="get_file",
                                 task_id="get_adl_file",
                                 get_logs=True,
                                 dag=dag
                                 )

I expect the pod to create a connection to datalake, but it is failing with the below error.

Kubernetes Pod logs

> Creating the Azure Datalake Client...
> ed2a3672-9d3b-456e-924f-cdf6e5f60ce8 - TokenRequest:Getting token with
> client credentials.
> /opt/conda/lib/python3.7/site-packages/sklearn/externals/joblib/__init__.py:15:
> DeprecationWarning: sklearn.externals.joblib is deprecated in 0.21 and
> will be removed in 0.23. Please import this functionality directly
> from joblib, which can be installed with: pip install joblib. If this
> warning is raised when loading pickled models, you may need to
> re-serialize those models with scikit-learn 0.21+.  
> warnings.warn(msg, category=DeprecationWarning)
> ed2a3672-9d3b-456e-924f-cdf6e5f60ce8 - OAuth2Client:Get Token request
> failed Traceback (most recent call last):   File
> "/opt/conda/lib/python3.7/site-packages/urllib3/connection.py", line
> 159, in _new_conn
>     (self._dns_host, self.port), self.timeout, extra_kw)   File "/opt/conda/lib/python3.7/site-packages/urllib3/util/connection.py",
> line 57, in create_connection
>     for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):   File "/opt/conda/lib/python3.7/socket.py", line
> 748, in getaddrinfo
>     for res in _socket.getaddrinfo(host, port, family, type, proto, flags): socket.gaierror: [Errno -3] Temporary failure in name
> resolution During handling of the above exception, another exception
> occurred: Traceback (most recent call last):   File
> "/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py",
> line 600, in urlopen
>     chunked=chunked)   File "/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py",
> line 343, in _make_request
>     self._validate_conn(conn)   File "/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py",
> line 839, in _validate_conn
>     conn.connect()   File "/opt/conda/lib/python3.7/site-packages/urllib3/connection.py", line
> 301, in connect
>     conn = self._new_conn()   File "/opt/conda/lib/python3.7/site-packages/urllib3/connection.py", line
> 168, in _new_conn
>     self, "Failed to establish a new connection: %s" % e) urllib3.exceptions.NewConnectionError:
> <urllib3.connection.VerifiedHTTPSConnection object at 0x7f0941ce8160>:
> Failed to establish a new connection: [Errno -3] Temporary failure in
> name resolution

Please help me to fix this connection error. Thanks

Answer 1

This seems like a very specific issue.

Try the “Slack” of Kubernetes, there are skilled users who are gladly to help out!