Python multiprocessing: object identifier unique across processes

Question

Suppose you want to run several processes in parallel (using multiprocessing, possibly on multiple separate machines, as in a cluster), where each process creates a list of new instances of a particular class. Then, you send all these lists back to the parent process, and you want to combine them. Now, can we index these instances by their object id? Can I expect the id's to uniquely identify the objects given that each object was generate on a separate process (possible a separate machine)?

In other words, does the id of an object survive the pickling required to send data between processes, or does the interpreter assign a fresh and unique id to the objects when unpickling them?

Answer 1

You asked, does the id of the object survive the pickling? The answer is no. The object is pickled and sent to another process, and a new object is created in that process with a new id. Results are sent back to the original process. The id does not survive… they are different objects. Id doesn't often survive pickling even in the same process… try obj2 = pickle.loads(pickle.dumps(object)) and see if obj2 is object … it's often not the case.

>>> import dill
>>>     
>>> class A(object):
...   pass
... 
>>> b = A()
>>> 
>>> id(b)
4473714832
>>> id(dill.loads(dill.dumps(b)))  
4486366032
>>> 

However, if you want to maintain an "id" as to understand which object is which, you can. Just add a id attribute that stores some id information (could be a simple number, such as the process "rank" (order), or could be something like a randomly generated hash, or something else… you pick). If you create this attribute ahead of time, and store the "id" there, it should maintain this information across a pickle . However, if you try to dynamically add an id attribute to any object, then pickle will "forget" that attribute was added, and the deserialized object will not have the attribute. Alternately, if you use an "advanced" serializer like dill , you can pickle a dynamically added attribute on a class instance or almost any object.