dask kubernetes aks（天藍色）虛擬節點

Question

使用下面的代碼可以在 azure aks 中創建一個 dask kubernetes 集群。

它使用遠程調度程序（ dask.config.set({"kubernetes.scheduler-service-type": "LoadBalancer"}) ）並且運行良好。

要使用虛擬節點，請取消注釋extra_pod_config=virtual_config行（遵循這個官方示例）。

它不起作用，出現以下錯誤：

ACI does not support providing args without specifying the command. Please supply both command and args to the pod spec.

這與傳遞containers: args: [dask-scheduler]

哪些containers: command:我應該提供哪些來解決這個問題？

謝謝

import dask
from dask.distributed import Client
from dask_kubernetes import KubeCluster, KubeConfig, make_pod_spec

image = "daskdev/dask"
cluster = "aks-cluster1"
dask.config.set({"kubernetes.scheduler-service-type": "LoadBalancer"})
dask.config.set({"distributed.comm.timeouts.connect": 180})
virtual_config = {
    "nodeSelector": {
        "kubernetes.io/role": "agent",
        "beta.kubernetes.io/os": "linux",
        "type": "virtual-kubelet",
    },
    "tolerations": [
        {"key": "virtual-kubelet.io/provider", "operator": "Exists"},
    ],
}

pod_spec = make_pod_spec(
    image=image,
    # extra_pod_config=virtual_config,
    memory_limit="2G",
    memory_request="2G",
    cpu_limit=1,
    cpu_request=1,
    threads_per_worker=1,  # same as cpu
)

# az aks get-credentials --name aks-cluster1 --resource-group resource_group1
# cp ~/.kube/config ./aksconfig.yaml
auth = KubeConfig(config_file="./aksconfig.yaml", context=cluster,)
cluster = KubeCluster(
    pod_spec, auth=auth, deploy_mode="remote", scheduler_service_wait_timeout=180
)
client = Client(cluster)

Answer 1

原因來自這個虛擬 kubelet 保護：在 pod 配置中，dask 使用args來啟動調度程序或工作程序，但沒有提供任何command 。

所以我明確地添加了入口點命令command_entrypoint_explicit並且它工作：成功創建了 pod。

第二個問題：網絡名稱解析。 工作人員無法通過網絡名稱連接到調度程序： tcp://{name}.{namespace}:{port}

雖然tcp://{name}.{namespace}.svc.cluster.local:{port}有效。 我在dask_kubernetes.core.Scheduler.start中編輯了它，它可以工作。

另一個選項是下面的virtual_config 。 下面的代碼是一個完整的解決方案。

import dask
from dask.distributed import Client
from dask_kubernetes import KubeCluster, KubeConfig, make_pod_spec

dask.config.set({"kubernetes.scheduler-service-type": "LoadBalancer"})
dask.config.set({"distributed.comm.timeouts.connect": 180})
image = "daskdev/dask"
cluster = "aks-cluster-prod3"
virtual_config = {
    "nodeSelector": {
        "kubernetes.io/role": "agent",
        "beta.kubernetes.io/os": "linux",
        "type": "virtual-kubelet",
    },
    "tolerations": [
        {"key": "virtual-kubelet.io/provider", "operator": "Exists"},
        {"key": "azure.com/aci", "effect": "NoSchedule"},
    ],
    "dnsConfig": {
        "options": [{"name": "ndots", "value": "5"}],
        "searches": [
            "default.svc.cluster.local",
            "svc.cluster.local",
            "cluster.local",
        ],
    },
}

# copied from: https://github.com/dask/dask-docker/blob/master/base/Dockerfile#L25
command_entrypoint_explicit = {
    "command": ["tini", "-g", "--", "/usr/bin/prepare.sh"],
}

pod_spec = make_pod_spec(
    image=image,
    extra_pod_config=virtual_config,
    extra_container_config=command_entrypoint_explicit,
    memory_limit="2G",
    memory_request="2G",
    cpu_limit=1,
    cpu_request=1,
    threads_per_worker=1,  # same as cpu
)

# az aks get-credentials --name aks-cluster1 --resource-group resource_group1
# cp ~/.kube/config ./aksconfig.yaml
auth = KubeConfig(config_file="./aksconfig.yaml", context=cluster,)
cluster = KubeCluster(
    pod_spec,
    auth=auth,
    deploy_mode="remote",
    scheduler_service_wait_timeout=180,
    n_workers=1,
)
client = Client(cluster)