如何腌制pyEphem對象以進行多處理？

Question

我正在嘗試計算衛星的一些值，數據生成需要很長時間，因此我想使用多處理來實現。

問題是我從pyEphem收到此錯誤， TypeError: can't pickle ephem.EarthSatellite objects 。 我要並行化的函數中沒有使用pyEphem對象。

這是我的代碼的示例文件（已最小化）。

這是我的主文件：

main.py

import ephem
import numpy
import math
import multiprocessing as mp
from SampleSats import Sats


GPS_Satellites = []

SFrames = 1
TLE = ["GPS BIIR-3  (PRN 11)",
       "1 25933U 99055A   18090.43292845 -.00000054  00000-0  00000+0 0  9994",
       "2 25933  51.8367  65.0783 0165007 100.2058 316.9161  2.00568927135407"]
# PRN TLE file from CelesTrak
GPS_Satellites.append(Sats(TLE))
Position = ephem.Observer()
Position.date = '2018/3/31 00:00'  # 1st January 2018 at 00:00 UTC
Position.lon, Position.lat = "36.845663", "-37.161123"   # Coordinates for desired Position

# Calculate Satellites
for Frames in range(SFrames):
    print("Generate Signals for Time: ", Position.date)
    for Sats in GPS_Satellites:  # par
        Sats.compute(Position)

        if ((float(repr(Sats.ephemeris.alt)) * 180 / math.pi) < 5) or (  # Calculate angle above horizon
                (float(repr(Sats.ephemeris.alt)) * 180 / math.pi) > 90):
            Sats.visible = 0
        else:
            Sats.visible = 1

    with mp.Pool() as pool:
        for value, obj in zip(pool.map(Sats.genSignal, GPS_Satellites), GPS_Satellites):
            obj.Signal = value

    Position.date = Position.date + 6*ephem.second  # 1 Subframe is 6 seconds long

這是我寫的Sats類：

sats.py ：

import ephem
import numpy

class Sats:
    """Save Satellites as Objects"""

    def __init__(self, tle):
        """:param tle: Two Line Element for ephemeris data also used to get PRN Number from name"""
        self.ephemeris = ephem.readtle(tle[0], tle[1], tle[2])
        self.visible = 1
        self.subframes = 0
        self.CAseq = [x for x in range(1023)]
        self.Out = []
        self.Signal = numpy.zeros(int(300*20*1023), dtype=numpy.int8)

    def compute(self, pos):
        self.ephemeris.compute(pos)
        self.Out.append(numpy.arange(0, 299, 1))
        self.subframes += 1

    def calcData(self, bit, prn):
         return (self.Out[self.subframes - 1].item(0)[0][bit] + self.CAseq[prn]) % 2

    def genSignal(self):
        if(self.visible == 1):
            for bit in range(300):  # 1 Subframe is 300 Bit long
                for x in range(20):  # The PRN Sequence reoccurs every ms -> 20 times per pit
                    for prn in range(1023):  # length of the prn sequence
                        self.Signal[bit*x*prn] = (-1 if (self.calcData(bit, prn))==0 else 1)
        else:
            self.Signal = numpy.zeros(300*20*1023)
        return self.Signal

追溯：

Traceback (most recent call last):
  File "C:/Users/PATH_TO_PROJECT/SampleTest.py", line 33, in <module>
    for value, obj in zip(pool.map(Sats.genSignal, GPS_Satellites), GPS_Satellites):
  File "C:\Program Files\Python36\lib\multiprocessing\pool.py", line 266, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "C:\Program Files\Python36\lib\multiprocessing\pool.py", line 644, in get
    raise self._value
  File "C:\Program Files\Python36\lib\multiprocessing\pool.py", line 424, in _handle_tasks
    put(task)
  File "C:\Program Files\Python36\lib\multiprocessing\connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "C:\Program Files\Python36\lib\multiprocessing\reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: can't pickle ephem.EarthSatellite objects

Answer 1

原因是這樣的...當您嘗試腌制一個函數時，它可能會嘗試腌制globals() ，因此您在全局命名空間中擁有的任何內容也會被腌制（以防萬一您的函數引用了globals()某些內容） globals() -是的，這是出乎意料的，但是事實就是這樣。 因此，一個簡單的解決方法是將您要腌制的功能隔離在另一個文件中–在這種情況下，將multiprocessing內容放入一個文件中，而將其他代碼放入另一個文件中...因此，對於pickler來說， globals()更少奮斗。 另一件事可能有用，那就是使用multiprocess而不是multiprocessing - multiprocess使用dill序列化器而不是pickle ，因此您更有可能序列化將通過Pool的工作程序發送的對象。