由於java.net.UnknownHostException，單節點kubernetes集群上的spark-examples作業失敗：kubernetes.default.svc

Question

由於java.net.UnknownHostException：kubernetes.default.svc，我將樣本spark（Spark代碼中提供的作業）提交到k8s集群。 如果你可以幫我解決這個問題，那將非常有幫助。

我的環境：

• Ubuntu 18.04 LTS amd64仿生圖像建於2019-06-17
• 2個vCPU 7.5 GB內存
• 雲服務：Google Coud Engine
• 僅限單個主節點（無工作節點）

如何重現我的問題：

$ kubectl cluster-info
Kubernetes master is running at https://10.128.0.10:6443
KubeDNS is running at https://10.128.0.10:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
$ bin/spark-submit \
    --master k8s://https://10.128.0.10:6443 \
    --deploy-mode cluster \
    --conf spark.executor.instances=3 \
    --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
    --conf spark.kubernetes.container.image=yohei1126/spark:v2.3.3 \
    --class org.apache.spark.examples.SparkPi \
    --name spark-pi \
    local:///opt/spark/examples/jars/spark-examples_2.11-2.3.3.jar

錯誤日志：

• KubeDNS正在運行，但名稱解析可能效果不佳。

    $ kubectl logs spark-pi-67ed1ddda23e32799371677bf1e795c4-driver
    ...
    2019-06-24 08:40:16 INFO  SparkContext:54 - Successfully stopped SparkContext
    Exception in thread "main" org.apache.spark.SparkException: External scheduler
    cannot be instantiated
    ...
    Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Operation:
    [get]  for kind: [Pod]  with name: [spark-pi-67ed1ddda23e32799371677bf1e795c4-driver]
    in namespace: [default]  failed.
    ...
    Caused by: java.net.UnknownHostException: kubernetes.default.svc: Try again

我如何在干凈的ubuntu上安裝k8s：

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

$ apt-get update && apt-get install -y apt-transport-https curl
$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
$ cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
$ apt-get update
$ apt-get install -y kubelet kubeadm kubectl
$ apt-mark hold kubelet kubeadm kubectl

我也安裝了Docker-ce，因為kubeadm需要它。

$ sudo apt update
$ sudo apt install -y \
    apt-transport-https \
    ca-certificates \
    curl \
    software-properties-common
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
$ sudo apt-key fingerprint 0EBFCD88
$ sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"
$ sudo apt update
$ sudo apt install -y docker-ce

我如何初始化集群：

• 指定的網絡地址為--pod-network-cidr。

    $ sudo kubeadm init --pod-network-cidr=10.128.0.0/20
    $ mkdir -p $HOME/.kube
    $ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    $ sudo chown $(id -u):$(id -g) $HOME/.kube/config

    $ sudo sysctl net.bridge.bridge-nf-call-iptables=1
    $ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
    $ kubectl taint nodes test-k8s node-role.kubernetes.io/master:NoSchedule-

我是如何創建docker鏡像的：

• 我使用了預先構建的Spark tar。

$ wget http://ftp.meisei-u.ac.jp/mirror/apache/dist/spark/spark-2.3.3/spark-2.3.3-bin-hadoop2.7.tgz
$ tar zxvf spark-2.3.3-bin-hadoop2.7.tgz
$ cd spark-2.3.3-bin-hadoop2.7
$ sudo bin/docker-image-tool.sh -r yohei1126 -t v2.3.3 build
$ sudo bin/docker-image-tool.sh -r yohei1126 -t v2.3.3 push

Answer 1

您能否根據本文檔[1]分享我們在Stackdriver Logging中與kube-system日志相關的條目？ 之前我看過同樣的問題，它與403的權限錯誤或IO超時有關。

您也可以嘗試重新創建nodepool。 它可以解決問題。

---

[1] https://cloud.google.com/monitoring/kubernetes-engine/legacy-stackdriver/logging

Answer 2

找到了一種使用minikube在Ubuntsu 18.04 LTS上構建單節點Kubernetes集群的方法。

• 機型：n1-standard-4（4個vCPU，15GB內存）
• 磁盤大小：30GB
• CPU平台：Intel Haswell
• Ubuntu 18.04 LTS
• Docker v18.09.7
• minikube v.1.2.0
• kubectl v1.15.0

安裝Docker

$ sudo apt-get update
$ sudo apt-get install -y \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
$ sudo apt-key fingerprint 0EBFCD88
$ sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"
$ sudo apt-get update
$ sudo apt-get install -y docker-ce docker-ce-cli containerd.io

安裝kubectl

$ sudo snap install kubectl --classic

安裝minikube

$ curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 \
  && chmod +x minikube
$ sudo install minikube /usr/local/bin

啟動kubernetes集群

• 注意：示例Spark作業需要8GB內存

$ sudo minikube start --vm-driver=none --cpu 4 --memory 8192
$ sudo mv /home/yohei/.kube /home/yohei/.minikube $HOME
$ sudo chown -R $USER $HOME/.kube $HOME/.minikube

為Spark作業創建Kubernetes服務帳戶

$ kubectl create serviceaccount spark
$ kubectl create clusterrolebinding spark-role --clusterrole=edit --serviceaccount=default:spark --namespace=default

檢查群集IP地址

$ kubectl cluster-info
Kubernetes master is running at https://10.128.0.11:6443
KubeDNS is running at https://10.128.0.11:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

運行Spark工作

$ sudo apt-get install openjdk-8-jdk -y
$ wget https://www-us.apache.org/dist/spark/spark-2.3.3/spark-2.3.3-bin-hadoop2.7.tgz
$ tar zxvf spark-2.3.3-bin-hadoop2.7.tgz
$ cd spark-2.3.3-bin-hadoop2.7
$ bin/spark-submit \
  --master k8s://https://10.128.0.11:6443 \
  --deploy-mode cluster \
  --conf spark.executor.instances=3 \
  --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
  --conf spark.kubernetes.container.image=yohei1126/spark:v2.3.3 \
  --class org.apache.spark.examples.SparkPi \
  --name spark-pi \
  local:///opt/spark/examples/jars/spark-examples_2.11-2.3.3.jar

檢查結果

2019-07-02 08:57:56 INFO  LoggingPodStatusWatcherImpl:54 - State changed, new state: 
         pod name: spark-pi-e39a8e8f7faf3c9fa861ae024e93b742-driver
         namespace: default
         labels: spark-app-selector -> spark-d0860239ee0f4118aeb8fee83bd00fa2, spark-role -> driver
         pod uid: 01e8f4c0-ae85-4252-92f7-11dbdd2e2b0d
         creation time: 2019-07-02T08:57:10Z
         service account name: spark
         volumes: spark-token-bnm7w
         node name: minikube
         start time: 2019-07-02T08:57:10Z
         container images: yohei1126/spark:v2.3.3
         phase: Succeeded
         status: [ContainerStatus(containerID=docker://c8c7584c7b704b8f2321943967f84d58267a9ca9d1e1852c2ac9eafb76816dc1, image=yohei1126/spark:v2.3.3, imageID=docker-pullable://yohei11
26/spark@sha256:d3524f24fe199dcb78fd3e1d640261e5337544aefa4aa302ac72523656fe2af1, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=s
park-kubernetes-driver, ready=false, restartCount=0, state=ContainerState(running=null, terminated=ContainerStateTerminated(containerID=docker://c8c7584c7b704b8f2321943967f84d58267a9ca
9d1e1852c2ac9eafb76816dc1, exitCode=0, finishedAt=Time(time=2019-07-02T08:57:56Z, additionalProperties={}), message=null, reason=Completed, signal=null, startedAt=Time(time=2019-07-02T
08:57:21Z, additionalProperties={}), additionalProperties={}), waiting=null, additionalProperties={}), additionalProperties={})]
2019-07-02 08:57:56 INFO  LoggingPodStatusWatcherImpl:54 - Container final statuses:
         Container name: spark-kubernetes-driver
         Container image: yohei1126/spark:v2.3.3
         Container state: Terminated
         Exit code: 0
2019-07-02 08:57:56 INFO  Client:54 - Application spark-pi finished.
2019-07-02 08:57:56 INFO  ShutdownHookManager:54 - Shutdown hook called
2019-07-02 08:57:56 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-9cbaba30-9277-4bc9-9c55-8acf78711e1d
$ kubectl get pods
NAME                                               READY   STATUS      RESTARTS   AGE
spark-pi-e39a8e8f7faf3c9fa861ae024e93b742-driver   0/1     Completed   0          43m
yohei_onishi@test-k8s:~$ kubectl logs spark-pi-e39a8e8f7faf3c9fa861ae024e93b742-driver
...
Pi is roughly 3.141395706978535
...