Kubernetes Pods status is always pending

Question

Please help me. I'm following this guide: https://pleasereleaseme.net/deploy-a-dockerized-asp-net-core-application-to-kubernetes-on-azure-using-a-vsts-ci-cd-pipeline-part-1/ . My docker images is successfully built and pushed to my private container registry and my CD pipeline is okay as well. But when I tried to check it using kubectl get pods the status is always pending, when I tried to use kubectl describe pod k8s-aspnetcore-deployment-64648bb5ff-fxg2k the message is:

Name:           k8s-aspnetcore-deployment-64648bb5ff-fxg2k
Namespace:      default
Node:           <none>
Labels:         app=k8s-aspnetcore
                pod-template-hash=2020466199
Annotations:    <none>
Status:         Pending
IP:
Controlled By:  ReplicaSet/k8s-aspnetcore-deployment-64648bb5ff
Containers:
  k8s-aspnetcore:
    Image:        mycontainerregistryb007.azurecr.io/k8saspnetcore:2033
    Port:         80/TCP
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-tr892 (ro)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  default-token-tr892:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-tr892
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  beta.kubernetes.io/os=windows
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age                From               Message
  ----     ------            ----               ----               -------
  Warning  FailedScheduling  3m (x57 over 19m)  default-scheduler  0/1 nodes are available: 1 MatchNodeSelector.

Here is for kubectl describe deployment:

Name:                   k8s-aspnetcore-deployment
Namespace:              default
CreationTimestamp:      Sat, 21 Jul 2018 13:41:52 +0000
Labels:                 app=k8s-aspnetcore
Annotations:            deployment.kubernetes.io/revision=2
Selector:               app=k8s-aspnetcore
Replicas:               1 desired | 1 updated | 1 total | 0 available | 1 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        5
RollingUpdateStrategy:  1 max unavailable, 1 max surge
Pod Template:
  Labels:  app=k8s-aspnetcore
  Containers:
   k8s-aspnetcore:
    Image:        mycontainerregistryb007.azurecr.io/k8saspnetcore:2033
    Port:         80/TCP
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
OldReplicaSets:  <none>
NewReplicaSet:   k8s-aspnetcore-deployment-64648bb5ff (1/1 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  26m   deployment-controller  Scaled up replica set k8s-aspnetcore-deployment-7f756cc78c to 1
  Normal  ScalingReplicaSet  26m   deployment-controller  Scaled up replica set k8s-aspnetcore-deployment-64648bb5ff to 1
  Normal  ScalingReplicaSet  26m   deployment-controller  Scaled down replica set k8s-aspnetcore-deployment-7f756cc78c to 0

Here is for kubectl describe service:

Name:                     k8s-aspnetcore-service
Namespace:                default
Labels:                   version=test
Annotations:              <none>
Selector:                 app=k8s-aspnetcore
Type:                     LoadBalancer
IP:                       10.0.26.188
LoadBalancer Ingress:     40.112.73.28
Port:                     <unset>  80/TCP
TargetPort:               80/TCP
NodePort:                 <unset>  30282/TCP
Endpoints:                <none>
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type    Reason                Age   From                Message
  ----    ------                ----  ----                -------
  Normal  EnsuringLoadBalancer  26m   service-controller  Ensuring load balancer
  Normal  EnsuredLoadBalancer   25m   service-controller  Ensured load balancer

I don't know if my setup for Kubernetes Cluster is wrong, but here is the script I used:

#!/bin/bash

azureSubscriptionId="xxxxxxx-xxxxx-xxxx-xxx-xxxxxxxx"
resourceGroup="k8sResourceGroup"
clusterName="k8sCluster"
location="northeurope"

# Useful if you have more than one Aure subscription
az account set --subscription $azureSubscriptionId

# Resource group for cluster - only availble in certain regions at time of writing
az group create --location $location --name $resourceGroup

# Create actual cluster
az aks create --resource-group $resourceGroup --name $clusterName --node-count 1 --generate-ssh-keys

# Creates a config file at ~/.kube on local machine to tell kubectl which cluster it should work with
az aks get-credentials --resource-group $resourceGroup --name $clusterName

Here is my deployment.yaml:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: k8s-aspnetcore-deployment
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  minReadySeconds: 5
  template:
    metadata:
      labels:
        app: k8s-aspnetcore
    spec:
      containers:
      - name: k8s-aspnetcore
        image: mycontainerregistryb007.azurecr.io/k8saspnetcore
        ports:
        - containerPort: 80
      imagePullSecrets:
        - name: k8ssecret
      nodeSelector:
        "beta.kubernetes.io/os": windows

Here is my service.yaml:

kind: Service
metadata:
  name: k8s-aspnetcore-service
  labels:
    version: test
spec:
  selector:
    app: k8s-aspnetcore
  ports:
  - port: 80
  type: LoadBalancer

Answer 1

The reason they are in Pending state is described in the message at the bottom of the describe pod output: MatchNodeSelector . That means Kubernetes didn't find a Node in your cluster that was able to fulfill the Node selection criteria specified in the PodSpec.

Specifically, it's very likely these lines:

nodeSelector:
    "beta.kubernetes.io/os": windows

Only a kubectl describe nodes would tell if there are any Nodes that could possibly fulfill that criteria

Answer 2

Just to build upon what @Matthew L Daniel has mentioned, the label-key "beta.kubernetes.io/os" is one of the default labels that come pre-populated in any Kubernetes Node , and which indicates the underlying OS.

By providing a label definition for the nodeSelector property of the Pod specification, you are telling the scheduler that you want the Pods managed by the Deployment object to only land on a Node which matches this criteria, meaning that they have to be placed in a Node running the Windows OS.

• But what if there's no Node matching such criteria ?

The following description command was ran on the cluster setup created by Minikube. The cluster is a single-node configuration, and the node name is minikube.

kubectl describe nodes minikube

The relevant part of description output was the following:

Name:               minikube
Roles:              master
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/hostname=minikube
                    node-role.kubernetes.io/master=
Annotations:        node.alpha.kubernetes.io/ttl=0
                    volumes.kubernetes.io/controller-managed-attach-detach=true
CreationTimestamp:  Wed, 11 Jul 2018 00:32:43 +0100
Taints:             <none>
Unschedulable:      false

As you can see on the Labels section there exists a definition for the label-key "beta.kubernetes.io/os" , but instead of being set to a value of "windows" is set to "linux" instead.