How do I pickle pyEphem objects for multiprocessing?

Question

I am trying to calculate some values of satellites, the data-generation takes quite long so I want to implement this using multiprocessing.

The problem is that I get this error from pyEphem, TypeError: can't pickle ephem.EarthSatellite objects . The pyEphem objects are not used in the functions that I want to parallelize.

This is an example file of my code (minimized).

This is my main file:

main.py

import ephem
import numpy
import math
import multiprocessing as mp
from SampleSats import Sats


GPS_Satellites = []

SFrames = 1
TLE = ["GPS BIIR-3  (PRN 11)",
       "1 25933U 99055A   18090.43292845 -.00000054  00000-0  00000+0 0  9994",
       "2 25933  51.8367  65.0783 0165007 100.2058 316.9161  2.00568927135407"]
# PRN TLE file from CelesTrak
GPS_Satellites.append(Sats(TLE))
Position = ephem.Observer()
Position.date = '2018/3/31 00:00'  # 1st January 2018 at 00:00 UTC
Position.lon, Position.lat = "36.845663", "-37.161123"   # Coordinates for desired Position

# Calculate Satellites
for Frames in range(SFrames):
    print("Generate Signals for Time: ", Position.date)
    for Sats in GPS_Satellites:  # par
        Sats.compute(Position)

        if ((float(repr(Sats.ephemeris.alt)) * 180 / math.pi) < 5) or (  # Calculate angle above horizon
                (float(repr(Sats.ephemeris.alt)) * 180 / math.pi) > 90):
            Sats.visible = 0
        else:
            Sats.visible = 1

    with mp.Pool() as pool:
        for value, obj in zip(pool.map(Sats.genSignal, GPS_Satellites), GPS_Satellites):
            obj.Signal = value

    Position.date = Position.date + 6*ephem.second  # 1 Subframe is 6 seconds long

This is the Sats class that i wrote:

sats.py :

import ephem
import numpy

class Sats:
    """Save Satellites as Objects"""

    def __init__(self, tle):
        """:param tle: Two Line Element for ephemeris data also used to get PRN Number from name"""
        self.ephemeris = ephem.readtle(tle[0], tle[1], tle[2])
        self.visible = 1
        self.subframes = 0
        self.CAseq = [x for x in range(1023)]
        self.Out = []
        self.Signal = numpy.zeros(int(300*20*1023), dtype=numpy.int8)

    def compute(self, pos):
        self.ephemeris.compute(pos)
        self.Out.append(numpy.arange(0, 299, 1))
        self.subframes += 1

    def calcData(self, bit, prn):
         return (self.Out[self.subframes - 1].item(0)[0][bit] + self.CAseq[prn]) % 2

    def genSignal(self):
        if(self.visible == 1):
            for bit in range(300):  # 1 Subframe is 300 Bit long
                for x in range(20):  # The PRN Sequence reoccurs every ms -> 20 times per pit
                    for prn in range(1023):  # length of the prn sequence
                        self.Signal[bit*x*prn] = (-1 if (self.calcData(bit, prn))==0 else 1)
        else:
            self.Signal = numpy.zeros(300*20*1023)
        return self.Signal

Traceback:

Traceback (most recent call last):
  File "C:/Users/PATH_TO_PROJECT/SampleTest.py", line 33, in <module>
    for value, obj in zip(pool.map(Sats.genSignal, GPS_Satellites), GPS_Satellites):
  File "C:\Program Files\Python36\lib\multiprocessing\pool.py", line 266, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "C:\Program Files\Python36\lib\multiprocessing\pool.py", line 644, in get
    raise self._value
  File "C:\Program Files\Python36\lib\multiprocessing\pool.py", line 424, in _handle_tasks
    put(task)
  File "C:\Program Files\Python36\lib\multiprocessing\connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "C:\Program Files\Python36\lib\multiprocessing\reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: can't pickle ephem.EarthSatellite objects

Answer 1

The reason is something like this... when you try to pickle a function, it can attempt to pickle globals() , so whatever you have in your global namespace is also pickled (just in case your function has a reference to something in globals() -- yes, that's unexpected, but that's how it is). So, an easy fix is to isolate the function you want to pickle in another file -- in this case, put the multiprocessing stuff in one file and the other code in another file... so there's less in globals() for the pickler to struggle with. Another thing that might help is to use multiprocess instead of multiprocessing -- multiprocess uses the dill serializer instead of pickle , so you have a better chance of serializing objects that will be sent across the workers in the Pool .