Kubernetes CronJob 未调度

Question

cronjob scheduled to run every minute. It ran as scheduled for several days, however 29hrs ago a bad deployment caused the jobs to fail due to ImagePullError, since then the cronjob does not schedule any additional jobs. The only jobs I can run are the ones that are manually created from the cronjob

spec:
  concurrencyPolicy: Forbid
  failedJobsHistoryLimit: 2
  jobTemplate:
    metadata:
      creationTimestamp: null
    spec:
      template:
        metadata:
          creationTimestamp: null
          labels:
            app.kubernetes.io/instance: acd-processor
            app.kubernetes.io/name: acd-processor
        spec:
          containers:
          - image: acdacr.azurecr.io/processor:964f395
            imagePullPolicy: IfNotPresent
            name: acd-processor
            resources: {}
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
          dnsPolicy: ClusterFirst
          imagePullSecrets:
          - name: acr-secret
          restartPolicy: OnFailure
          schedulerName: default-scheduler
          securityContext: {}
          terminationGracePeriodSeconds: 30
  schedule: '* * * * *'
  successfulJobsHistoryLimit: 1
  suspend: false
status:
  lastScheduleTime: "2022-01-27T21:07:00Z"

NAME                 COMPLETIONS   DURATION   AGE
job.batch/test-job   1/1           3s         20m

NAME                                        SCHEDULE    SUSPEND   ACTIVE   LAST SCHEDULE   AGE
cronjob.batch/acd-processor-cronjob   * * * * *   False     0        29h             8d

pod/test-job-ftx5w                              0/1     Completed   0          20m

test-job is manually created from the cronjob

The cronjob is not suspended, and there are 0 active jobs so it should work even though concurrency is disabled.

Why does the cronjob not continue to schedule as it's configured?

Answer 1

An almost identical question was asked at Kubernetes Not Scheduling CronJob

I'll copy-paste my answer from there as it has some things for you to check

A couple of other things you can check:

1. Do you have any cron pods with a "failed" status? If you do, check those pods for why.
2. Did it used to work and then suddenly stop?
3. Does the cronjob resource have anything in the events? kubectl describe cronjob health-status-cron -n tango
4. Does the code your cron runs take > 1 minute to complete? If so, your schedule is too aggressive, and you might want to loosen the schedule
5. The cronjob controller also has some limitations you may want to check: https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#cron-job-limitations . Specifically the concept of "missed jobs". If the cronjob controller "misses" scheduling 100 or more jobs, it will "freeze" the job and not schedule it anymore. Do you scale down the cluster or similar when it is not in use?
6. Do you have any custom/third-party webhooks or plugins installed in the cluster? These can interfere with pod creation.
7. Do you have any jobs created in the namespace? kubectl get jobs -n tango If you find a ton of job objects, check them to see why they did not generate pods.

I encountered a somewhat similar issue in 2020 (writeup has a link to the issue I raised in the Kubernetes project itself): https://blenderfox.com/2020/08/07/the-snowball-effect-in-kubernetes/