can't pickle _thread.RLock objects when using a webservice

Question

I am using python 3.6

I am trying to use multiprocessing from inside a class method shown below by the name SubmitJobsUsingMultiProcessing() which further calls another class method in turn.

I keep running into this error : Type Error : can't pickle _thread.RLock objects.

I have no idea what this means. I have a suspicion that the below line trying to establish a connection to a webserver API might be responsible but I am all at sea to understand why.

I am not a proper programmer(code as a part of a portfolio modeling team) so if this is an obvious question please pardon my ignorance and many thanks in advance.

import multiprocessing as mp,functools

def SubmitJobsUsingMultiProcessing(self,PartitionsOfAnalysisDates,PickleTheJobIdsDict = True):
    if (self.ExportSetResult == "SUCCESS"):
        NumPools = mp.cpu_count()
        PoolObj =  mp.Pool(NumPools)   
        userId,clientId,password,expSetName = self.userId , self.clientId , self.password , self.expSetName
        PartialFunctor = functools.partial(self.SubmitJobsAsOfDate,userId = userId,clientId = clientId,password = password,expSetName = expSetName)
        Result = PoolObj.map(self.SubmitJobsAsOfDate, PartitionsOfAnalysisDates)
        BatchJobIDs = OrderedDict((key, val) for Dct in Result for key, val in Dct.items())
        f_pickle = open(self.JobIdPickleFileName, 'wb')
        pickle.dump(BatchJobIDs, f_pickle, -1)
        f_pickle.close()


 def SubmitJobsAsOfDate(self,ListOfDatesForBatchJobs,userId,clientId,password,expSetName):

    client = Client(self.url, proxy=self.proxysettings)
    if (self.ExportSetResult != "SUCCESS"):
        print("The export set creation was not successful...exiting")
        sys.exit()

    BatchJobIDs = OrderedDict()
    NumJobsSubmitted = 0
    CurrentProcessID = mp.current_process()

    for AnalysisDate in ListOfDatesForBatchJobs:
        jobName = "Foo_" + str(AnalysisDate)
        print('Sending job from process : ', CurrentProcessID, ' : ', jobName)
        jobId = client.service.SubmitExportJob(userId,clientId,password,expSetName, AnalysisDate, jobName, False)
        BatchJobIDs[AnalysisDate] = jobId
        NumJobsSubmitted += 1

        'Sleep for 30 secs every 100 jobs'
        if (NumJobsSubmitted % 100 == 0):
            print('100 jobs have been submitted thus far from process : ', CurrentProcessID,'---Sleeping for 30 secs to avoid the SSL time out error')
            time.sleep(30)
    self.BatchJobIDs = BatchJobIDs
    return BatchJobIDs

Below is the trace ::

    Traceback (most recent call last):
  File "C:\Program Files\JetBrains\PyCharm 2017.2.3\helpers\pydev\pydevd.py", line 1599, in <module>
    globals = debugger.run(setup['file'], None, None, is_module)
  File "C:\Program Files\JetBrains\PyCharm 2017.2.3\helpers\pydev\pydevd.py", line 1026, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "C:\Program Files\JetBrains\PyCharm 2017.2.3\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "C:/Users/trpff85/PycharmProjects/QuantEcon/BDTAPIMultiProcUsingPathos.py", line 289, in <module>
    BDTProcessObj.SubmitJobsUsingMultiProcessing(Partitions)
  File "C:/Users/trpff85/PycharmProjects/QuantEcon/BDTAPIMultiProcUsingPathos.py", line 190, in SubmitJobsUsingMultiProcessing
    Result = PoolObj.map(self.SubmitJobsAsOfDate, PartitionsOfAnalysisDates)
  File "C:\Users\trpff85\AppData\Local\Continuum\anaconda3\lib\multiprocessing\pool.py", line 266, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "C:\Users\trpff85\AppData\Local\Continuum\anaconda3\lib\multiprocessing\pool.py", line 644, in get
    raise self._value
  File "C:\Users\trpff85\AppData\Local\Continuum\anaconda3\lib\multiprocessing\pool.py", line 424, in _handle_tasks
    put(task)
  File "C:\Users\trpff85\AppData\Local\Continuum\anaconda3\lib\multiprocessing\connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "C:\Users\trpff85\AppData\Local\Continuum\anaconda3\lib\multiprocessing\reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: can't pickle _thread.RLock objects

Answer 1

I am struggling with a similar problem. There was a bug in <=3.5 whereby _thread.RLock objects did not raise an error when pickled (They cannot be) For the Pool object to work, a function and arguments must be passed to it from the main process and this relies on pickling (pickling is a means of serialising objects) In my case the RLock object is somewhere in the logging module. I suspect your code will work fine on 3.5. Good luck. See this bug resolution.