使用 C 绑定导入代码的 Pickle 模块

Question

I have a class I want to pickle using this function:

def _pickle(self):
    """Pickle model instance.
    """
    fpath = f'{directory}/model.pickle'
    with open(fpath, 'wb') as f:
        pickle.dump(self, f, pickle.HIGHEST_PROTOCOL)

However, in my file I import a library, PyPolyaGamma . It's a Python wrapper for some C code that does fast sampling from the Polya-gamma distribution. When I try to pickle my class, I get this error:

Traceback (most recent call last):
  File "fit_model.py", line 262, in <module>
    model.fit(Y)
  File "/Users/gwg/projects/cdnlvm/kl_gplvmbase.py", line 97, in fit
    self._plot_and_print(t)
  File "/Users/gwg/projects/cdnlvm/kl_nbgplvm.py", line 456, in _plot_and_print
    self._pickle()
  File "/Users/gwg/projects/cdnlvm/kl_gplvmbase.py", line 352, in _pickle
    pickle.dump(self_, f, pickle.HIGHEST_PROTOCOL)
  File "stringsource", line 2, in pypolyagamma.pypolyagamma.PyPolyaGamma.__reduce_cython__
TypeError: no default __reduce__ due to non-trivial __cinit__

My understanding is that Python does not know how to serialize the PyPolyaGamma code, probably because of the dependency on C. What is the correct way to handle this?

---

Here is a minimal and complete example, if you're willing to install PyPolyaGamma:

# pickle_test.py

import pickle
from   pypolyagamma import PyPolyaGamma

class Model:
    def __init__(self):
        self.pg = PyPolyaGamma()

model = Model()
with open('test.pickle', 'wb+') as f:
    pickle.dump(model, f)

This will output:

Traceback (most recent call last):
  File "pickle_test.py", line 15, in <module>
    pickle.dump(model, f)
  File "stringsource", line 2, in pypolyagamma.pypolyagamma.PyPolyaGamma.__reduce_cython__
TypeError: no default __reduce__ due to non-trivial __cinit__

Answer 1

Unfortunately it looks like that pypolyagamma library you use doesn't currently support pickling/unpickling those objects, so in a word: you can't pickle them without modifying the library.

Furthermore, since it wraps a C++ class, the pickler/unpickler isn't trivial; it'd need to know what state from that wrapped class is necessary for the object to be in the same state after unpickling.

Answer 2

As an alternative to a wrapper and if you don't mind using a functional interface, then I would recommend using polyagamma instead. This resolves issues with pickling. Its a python package for Polya-Gamma variable sampling written in C. It is flexible and allows using different sampling methods and has an API that resembles Numpy's. You can install the latest version via pip install -U polyagamma .