Python pickling error: TypeError: object pickle not returning list

Question

I am aware if this very old question Python pickling error: TypeError: object pickle not returning list. Issue with numpy? but the only answer given is rather obscure.

This is the code that reproduces the issue in Python 3.6.

import pickle
from astroquery.irsa import Irsa
from astropy import units as u

# Query region.
table = Irsa.query_region("m31", catalog="fp_psc", spatial="Cone",
                          radius=.5 * u.arcmin)

# Dump table.
with open('table.pkl', 'wb') as f:
    pickle.dump(table, f)

# This is where the issue appears.
with open('table.pkl', 'rb') as f:
    table = pickle.load(f)

When trying to load the pickled data, it fails with:

Traceback (most recent call last):
  File "/home/gabriel/Descargas/test.py", line 17, in <module>
    table2 = pickle.load(f)
  File "/home/gabriel/anaconda3/envs/cat-match/lib/python3.6/site-packages/astropy/table/column.py", line 238, in __setstate__
    super_class.__setstate__(self, state)
  File "/home/gabriel/anaconda3/envs/cat-match/lib/python3.6/site-packages/numpy/ma/core.py", line 5869, in __setstate__
    super(MaskedArray, self).__setstate__((shp, typ, isf, raw))
TypeError: object pickle not returning list

How can I get around this?

---

Using conda you can install the requirements in a Python 3 environment with:

conda install astropy
conda install -c astropy astroquery

Answer 1

The linked answer mentions numpy masked arrays causing trouble when unpickling them. Given that astropy tables may indeed have masked entries (not every object or position may have a value for, eg, every band or whatever relevant columns), this could indeed be the source of your problem: masked arrays.

The numpy issue trickers shows that problem as well: Masked array with object dtype doesn't unpickle . And that appears to be solved in pull request 8122 .

Scouring the release notes for Numpy 1.12.0 shows this pull request in there. So unless you're already using numpy 1.12, it may be your worth to upgrade numpy.