如何在 Cloud Composer 中通过 API 访问 Kubernetes 服务

Question

I am trying to send an API request to a Kubernetes Service using Cloud Composer. I have the following snippet:

def fetchToken():
    ...


def start_application():
keycloak_token = fetchToken()
url = "http://<name>.svc.cluster.local/tasks"

with open("/home/airflow/gcs/dags/src/somefile.json") as f:
    d = json.load(f)

response = requests.put(
    url,
    headers={
        "Authorization": f"Bearer {keycloak_token}",
        "Content-type": "application/json",
    },
    data=json.dumps(d),
)

However, this gives me the following error:

(HTTPConnectionPool(host='<name>.svc.cluster.local', port=80): Max retries exceeded with url: /tasks:8080 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f03e0d97d60>: Failed to establish a new connection: [Errno -2] Name or service not known')); 25941)

Seems like I cannot access as the URL is "svc.cluster.local". Any idea how to access the endpoint?

Thanks!

Answer 1

Cloud composer is a google service independent from GKE, if you don't want to expose your api to public, you can create an internal load balancer to expose the service in the vpc, so by using the same vpc on cloud composer and GKE, your cloud composer dags could access the API in GKE.

There are many other options and tools to achieve this.