Joblib錯誤：TypeError：無法腌制_thread.lock對象

Question

我無法使用需要一個numpy數組，訓練有素的Keras模型列表和字符串列表作為參數的函數來運行joblib。

我嘗試將參數創建為namedtuple或具有不變屬性的類。 有任何想法嗎 ？

Params = collections.namedtuple('Params',['inputs','y_list','trained_models'])
p = Params(inputs, y_list, trained_models)

要么

class Params:
    def __init__(self, inputs, mq_list,trained_models):
        super(Params , self).__setattr__("inputs", inputs)
        super(Params , self).__setattr__("y_list", y_list)
        super(Params , self).__setattr__("trained_models", trained_models)

我喜歡並行運行的功能：

def predict(params):
    inputs = params.inputs
    y_list = params.y_list
    trained_models = params.trained_models

    # process and vectorize inputs
    X= new_X(inputs)
    X_vect= vect.transform(X)    
    predictions = dict()  

    for y in y_list:
        y_field = trained_models[y].predict(X_vect)
        # evaluate model
        if y_field[0] > 0.05:
            return None, None

        predictions[y] = y_field[0]

    return X, predictions

並行調用函數：

r= Parallel(n_jobs=4, verbose=5)(
    delayed(predict)(p)
    for c in range(100))

錯誤：

TypeErrorTraceback (most recent call last) <timed exec> in <module>()

~/.conda/envs/mlgpu/lib/python3.6/site-packages/joblib/parallel.py in
__call__(self, iterable)
    787                 # consumption.
    788                 self._iterating = False
--> 789             self.retrieve()
    790             # Make sure that we get a last message telling us we are done
    791             elapsed_time = time.time() - self._start_time

~/.conda/envs/mlgpu/lib/python3.6/site-packages/joblib/parallel.py in retrieve(self)
    697             try:
    698                 if getattr(self._backend, 'supports_timeout', False):
--> 699                     self._output.extend(job.get(timeout=self.timeout))
    700                 else:
    701                     self._output.extend(job.get())

~/.conda/envs/mlgpu/lib/python3.6/multiprocessing/pool.py in get(self, timeout)
    642             return self._value
    643         else:
--> 644             raise self._value
    645 
    646     def _set(self, i, obj):

~/.conda/envs/mlgpu/lib/python3.6/multiprocessing/pool.py in
_handle_tasks(taskqueue, put, outqueue, pool, cache)
    422                         break
    423                     try:
--> 424                         put(task)
    425                     except Exception as e:
    426                         job, idx = task[:2]

~/.conda/envs/mlgpu/lib/python3.6/site-packages/joblib/pool.py in send(obj)
    369             def send(obj):
    370                 buffer = BytesIO()
--> 371                 CustomizablePickler(buffer, self._reducers).dump(obj)
    372                 self._writer.send_bytes(buffer.getvalue())
    373             self._send = send

TypeError: can't pickle _thread.lock objects

Answer 1

您應該創建自己的類，因為您不知道函數collections.namedtuple是否具有不可拾取的部分。

幾個月前，我遇到了類似的問題，當時我在類中添加了lambda函數以將其作為參數傳遞。 但是由於lambda函數不可（通過pickle包進行選擇），因此會產生錯誤。