In a Kubernetes watch operation for list of pod, are resourceVersion guaranteed to be monotonically increasing?

Question

From Kube.netes API Concepts > Efficient detection of changes :

When retrieving a collection of resources (either namespace or cluster scoped), the response from the API server contains a resourceVersion value. The client can use that resourceVersion to initiate a watch against the API server. When you send a watch request, the API server responds with a stream of changes. These changes itemize the outcome of operations (such as create, delete, and update) that occurred after the resourceVersion you specified as a parameter to the watch request. The overall watch mechanism allows a client to fetch the current state and then subscribe to subsequent changes, without missing any events.

When I tried a watch operation (using kube.netes python client) I get a stream of kube.netes events, the events themselves do not have a resourceVersion , the object inside the event ( kind: Pod ) do have resourceVersion .

from kubernetes import client,config,watch
config.load_kube_config(context='eks-prod')
v1 = client.CoreV1Api()
watcher = watch.Watch()
namespace = 'my-namespace'
last_resource_version=0
for i in  watcher.stream(v1.list_namespaced_pod, namespace, resource_version=last_resource_version, timeout_seconds=5):
   print(i['object'].metadata.resource_version)
   last_resource_version = i['object'].metadata.resource_version

The resource version are output in the order they are received and they are not monotonically increasing at least in the initial batch of events:

380744163
380641499
380710458
380775853
380525082
381044514
380676150
380641735
380566799
380806984
380566585
380710721
378885571
380524650
380710218
380806798
373502190
380566206
381044372
380524615
380676624
380806573
380775663
380605904
380743917
380606153
380676388
380744368
380641258
380775416
380606397

But can I assume that if this watch is disconnected I can safely resume from the highest resource version I've seen? In the above case, can I safely resume from 381044514 (the highest) without missing events?

Answer 1

From Resource Version Semantics

You must not assume resource versions are numeric or collatable. API clients may only compare two resource versions for equality (this means that you must not compare resource versions for greater-than or less-than relationships).

So in principle no you can't use the "highest" resource version because they are not really numeric or sortable. The best you can do is use the latest resourceVersion that you received as is, verbatim. And be prepared to get a resource too old that you are supposed to handle by retrying without specifying a resource version , in that case you must also handle the case where you will likely receive some events more than once.

This scenario where the resourceVersion in the last event received is not the actual latest/most recent is easily reproducible in EKS 1.21 where the initial response to the watch will return the events in more or less random order. If I send two watch requests simultaneously I'll get the same 30 events but in different order.