将我的服务部署到 azure kubernetes 服务时，总是出现错误“部署‘xxx’超出其进度截止日期”

Question

I am newer to AKS, and I am setting up an CI/CD using AKS. I created an azure kubernetes service with two node pool, one is the default one (nodepool1, with linux OS), the another one is windows os node pool (wdpool) which I want to deploy to.

my kubernetes node pools: My azure kubernetes node pools snapshot

I saw my repository image is pushed successfully: 'xap' repository is pushed successfully snapshot

the kubeevents error as following, how should I fix it ?

Error: failed to start container "xap": Error response from daemon: hcsshim::CreateComputeSystem xap: The container operating system does not match the host operating system.
"

1. my deployment.yml

apiVersion : apps/v1beta1
kind: Deployment
metadata:
  name: xap 
  namespace: kube-public
spec:
  progressDeadlineSeconds: 900

  replicas: 1
  template:
    metadata:
      labels:
        app: xap 
    spec:
      nodeSelector:
        agentpool: wdpool
        beta.kubernetes.io/os: windows
      containers:
        - name: xap 
          image: xapcontainerregistry.azurecr.io/xap

          ports:
          - containerPort: 80

2. my service.yml

apiVersion: v1
kind: Service
metadata:
    name: xap
spec:
    type: LoadBalancer
    ports:
    - port: 80
    selector:
        app: xap

3. My deployment task in build yml:

# Deploy to Kubernetes - Review app with Azure DevSpaces
# Build and push image to Azure Container Registry; Deploy to Azure Kuberentes Services
# https://docs.microsoft.com/azure/devops/pipelines/languages/docker

trigger:
- users/jasca/xapaks

resources:
- repo: self

variables:

  # Container registry service connection established during pipeline creation
  dockerRegistryServiceConnection: '49c3dcce-5bc7-4ea4-90fe-4bb701a497e8'
  imageRepository: 'xap'
  containerRegistry: 'xapcontainerregistry.azurecr.io'
  dockerfilePath: '**/Dockerfile'
  tag: '$(Build.BuildId)'
  imagePullSecret: 'xapcontainerregistry4671248d-auth'

  # Agent VM image name
  vmImageName: 'windows-latest'

  # Azure Service connection
  azureConnection: '4e998ed5-bbcb-4495-b19e-1ad6ca32bbe1'

  # Name of the new namespace being created to deploy the PR changes.
  k8sNamespaceForPR: '$(system.pullRequest.sourceBranch)'
  buildConfiguration: 'Debug'

stages:
- stage: Build
  displayName: Build WebContainerHost
  jobs:  
  - job: Build
    displayName: Build
    pool:
#      vmImage: $(vmImageName)
       name: XAP_AKS_Pool
#      name: XapOuterloopProductionOnly
#      demands:
#        - Agent.ComputerName -equals xapagent4
    steps:

    # ignore build tasks, succeed.

- stage: Deploy
  displayName: Deploy stage
  dependsOn: Build

  jobs:
  - deployment: Deploy
    condition: and(succeeded(), not(startsWith(variables['Build.SourceBranch'], 'refs/pull/')))
    displayName: Deploy
    pool:
      vmImage: $(vmImageName)
    environment: 'xapplatform.kube-public'
    strategy:
      runOnce:
        deploy:
          steps:
          - task: KubernetesManifest@0
            displayName: Create imagePullSecret
            inputs:
              action: createSecret
              secretName: $(imagePullSecret)
              dockerRegistryEndpoint: $(dockerRegistryServiceConnection)

          - task: KubernetesManifest@0
            displayName: Deploy to Kubernetes cluster
            inputs:
              rolloutStatusTimeout: 1800

              action: deploy 
              manifests: |
                $(Pipeline.Workspace)/manifests/deployment.yml
                $(Pipeline.Workspace)/manifests/service.yml
              imagePullSecrets: |
                $(imagePullSecret)
              containers: |
                $(containerRegistry)/$(imageRepository):$(tag)
            timeoutInMinutes: 40



  - deployment: DeployPullRequest
    displayName: Deploy Pull request
    condition: and(succeeded(), startsWith(variables['Build.SourceBranch'], 'refs/pull/'))
    pool:
      vmImage: $(vmImageName)

    environment: 'xapplatform.$(k8sNamespaceForPR)'
    strategy:
      runOnce:
        deploy:
          steps:
          - reviewApp: kube-public

          - task: Kubernetes@1
            displayName: 'Create a new namespace for the pull request'
            inputs:
              command: apply
              useConfigurationFile: true
              inline: '{ "kind": "Namespace", "apiVersion": "v1", "metadata": { "name": "$(k8sNamespaceForPR)" }}'

          - task: KubernetesManifest@0
            displayName: Create imagePullSecret
            inputs:
              action: createSecret
              secretName: $(imagePullSecret)
              namespace: $(k8sNamespaceForPR)
              dockerRegistryEndpoint: $(dockerRegistryServiceConnection)

          - task: KubernetesManifest@0
            displayName: Deploy to the new namespace in the Kubernetes cluster
            inputs:
              action: deploy
              namespace: $(k8sNamespaceForPR)
              manifests: |
                $(Pipeline.Workspace)/manifests/deployment.yml
                $(Pipeline.Workspace)/manifests/service.yml
              imagePullSecrets: |
                $(imagePullSecret)
              containers: |
                $(containerRegistry)/$(imageRepository):$(tag)

          - task: Kubernetes@1
            name: get
            displayName: 'Get services in the new namespace'
            continueOnError: true
            inputs:
              command: get
              namespace: $(k8sNamespaceForPR)
              arguments: svc
              outputFormat: jsonpath='http://{.items[0].status.loadBalancer.ingress[0].ip}:{.items[0].spec.ports[0].port}'

          # Getting the IP of the deployed service and writing it to a variable for posing comment
          - script: |
              url="$(get.KubectlOutput)"
              message="Your review app has been deployed"
              if [ ! -z "$url" -a "$url" != "http://:" ] 
              then
                message="${message} and is available at $url.<br><br>[Learn More](https://aka.ms/testwithreviewapps) about how to test and provide feedback for the app."
              fi
              echo "##vso[task.setvariable variable=GITHUB_COMMENT]$message"

4. My createPullSecret task succeed:

   ##[section]Starting: Create imagePullSecret
==============================================================================
Task         : Deploy to Kubernetes
Description  : Use Kubernetes manifest files to deploy to clusters or even bake the manifest files to be used for deployments using Helm charts
Version      : 0.165.4
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/deploy/kubernetes-manifest
==============================================================================
==============================================================================
            Kubectl Client Version: v1.17.1
            Kubectl Server Version: v1.14.8
==============================================================================
[command]C:\ProgramData\Chocolatey\bin\kubectl.exe delete secret xapcontainerregistry4671248d-auth --namespace kube-public
secret "xapcontainerregistry4671248d-auth" deleted
[command]C:\ProgramData\Chocolatey\bin\kubectl.exe create secret docker-registry xapcontainerregistry4671248d-auth --docker-username *** --docker-password *** --docker-server *** --docker-email ServicePrincipal@AzureRM --namespace kube-public
secret/xapcontainerregistry4671248d-auth created
##[section]Finishing: Create imagePullSecret

5. My deploy to kubernetes cluster task log:

##[section]Starting: Deploy to Kubernetes cluster
==============================================================================
Task         : Deploy to Kubernetes
Description  : Use Kubernetes manifest files to deploy to clusters or even bake the manifest files to be used for deployments using Helm charts
Version      : 0.165.4
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/deploy/kubernetes-manifest
==============================================================================
==============================================================================
            Kubectl Client Version: v1.17.1
            Kubectl Server Version: v1.14.8
==============================================================================
[command]C:\ProgramData\Chocolatey\bin\kubectl.exe apply -f d:\a\_temp\Deployment_xap_1583635691677,d:\a\_temp\Service_xap_1583635691677 --namespace kube-public
deployment.apps/xap configured
service/xap unchanged
[command]C:\ProgramData\Chocolatey\bin\kubectl.exe rollout status Deployment/xap --timeout 1800s --namespace kube-public
error: deployment "xap" exceeded its progress deadline
Waiting for deployment "xap" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment spec update to be observed...
Waiting for deployment spec update to be observed...
Waiting for deployment "xap" rollout to finish: 1 old replicas are pending termination...
##[error]Error: error: deployment "xap" exceeded its progress deadline
[command]C:\ProgramData\Chocolatey\bin\kubectl.exe describe Deployment xap --namespace kube-public
Name:                   xap
Namespace:              kube-public
CreationTimestamp:      Fri, 06 Mar 2020 14:16:37 +0000
Labels:                 app=xap
Annotations:            deployment.kubernetes.io/revision: 15
                        kubectl.kubernetes.io/last-applied-configuration:
                          {"apiVersion":"apps/v1beta1","kind":"Deployment","metadata":{"annotations":{},"name":"xap","namespace":"kube-public"},"spec":{"progressDea...
Selector:               app=xap
Replicas:               1 desired | 1 updated | 2 total | 0 available | 2 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=xap
  Containers:
   xap:
    Image:        ***/xap
    Port:         80/TCP
    Host Port:    0/TCP
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      False   MinimumReplicasUnavailable
  Progressing    False   ProgressDeadlineExceeded
OldReplicaSets:  xap-5b95494dd7 (1/1 replicas created)
NewReplicaSet:   xap-79cc65ddf9 (1/1 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  36m   deployment-controller  Scaled down replica set xap-668784656f to 0
  Normal  ScalingReplicaSet  36m   deployment-controller  Scaled up replica set xap-694857dcd8 to 1
  Normal  ScalingReplicaSet  36m   deployment-controller  Scaled down replica set xap-67fd46c7b5 to 0
  Normal  ScalingReplicaSet  36m   deployment-controller  Scaled up replica set xap-79cc65ddf9 to 1
  Normal  ScalingReplicaSet  10m   deployment-controller  Scaled down replica set xap-694857dcd8 to 0
  Normal  ScalingReplicaSet  10m   deployment-controller  Scaled up replica set xap-5b95494dd7 to 1
For more information, go to https://msasg.visualstudio.com/Bing_and_IPG/_environments/75/providers/kubernetes/12
[command]C:\ProgramData\Chocolatey\bin\kubectl.exe get service/xap -o json --namespace kube-public
{
    "apiVersion": "v1",
    "kind": "Service",
    "metadata": {
        "annotations": {
            "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"name\":\"xap\",\"namespace\":\"kube-public\"},\"spec\":{\"ports\":[{\"port\":80}],\"selector\":{\"app\":\"xap\"},\"type\":\"LoadBalancer\"}}\n"
        },
        "creationTimestamp": "2020-03-06T14:16:38Z",
        "name": "xap",
        "namespace": "kube-public",
        "resourceVersion": "1743607",
        "selfLink": "/api/v1/namespaces/kube-public/services/xap",
        "uid": "17d862ed-5fb5-11ea-ba44-82c49be4b313"
    },
    "spec": {
        "clusterIP": "10.0.76.15",
        "externalTrafficPolicy": "Cluster",
        "ports": [
            {
                "nodePort": 30742,
                "port": 80,
                "protocol": "TCP",
                "targetPort": 80
            }
        ],
        "selector": {
            "app": "xap"
        },
        "sessionAffinity": "None",
        "type": "LoadBalancer"
    },
    "status": {
        "loadBalancer": {
            "ingress": [
                {
                    "ip": "52.139.245.51"
                }
            ]
        }
    }
}
service xap external IP is 52.139.245.51
##[error]Rollout status check failed.
##[section]Finishing: Deploy to Kubernetes cluster

Answer 1

According @Arghya-Sadhu's suggestion, I checked the logs in Azure logs, the table of "KubeEvent", it says that

failed to start container "xap": Error response from daemon: hcsshim::CreateComputeSystem xap: The container operating system does not match the host operating system. 

The above error is due to my docker building agent is using a Windows Server 2016, and the azure container systems is Windows Server 2019, it is not match... I upgraded my building agent machine OS to Windows Server 2019, the issue resolved and deployed to Kubernets successfully.