[英]Keras model: TypeError: can't pickle _thread.lock objects
I have trouble using a trained Keras model in PySpark. 我在PySpark中使用训练有素的Keras模型遇到麻烦。 The following versions of libraries are used:
使用以下版本的库:
tensorflow==1.1.0
h5py==2.7.0
keras==2.0.4
Also, I use Spark 2.4.0. 另外,我使用Spark 2.4.0。
from pyspark.sql import SparkSession
import pyspark.sql.functions as func
from keras.models import load_model
spark = SparkSession \
.builder \
.appName("Test") \
.master("local[2]") \
.getOrCreate()
my_model = load_model("my_model.h5")
spark.sparkContext.addFile("my_model.h5")
my_model_bcast = spark.sparkContext.broadcast(my_model)
# ...
get_prediction_udf = func.udf(get_prediction, IntegerType())
ds = ds\
.withColumn("predicted_value", get_prediction_udf(my_model_bcast,
func.col("col1"),
func.col("col2"))))
The function get_prediction
looks as follows (a simplified code): 函数
get_prediction
如下所示(简化代码):
def get_prediction(my_model_bcast, col1, col2):
cur_state = np.array([col1,col2])
state = cur_state.reshape(1,2)
ynew = my_model_bcast.predict(state)
return np.argmax(ynew[0])
The following error is triggered by the line my_model_bcast = spark.sparkContext.broadcast(my_model)
: 以下错误由行
my_model_bcast = spark.sparkContext.broadcast(my_model)
触发:
File "/usr/local/spark-2.4.0-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/broadcast.py", line 110, in dump
pickle.dump(value, f, 2)
TypeError: can't pickle _thread.lock objects
I was reading similar threads in order to find a solution. 我正在阅读类似的主题,以便找到解决方案。 As far as I understand,
keras
is not supporting applying pickle
. 据我了解,
keras
不支持应用pickle
。 But in this case how can I make predictions in PySpark using a trained model? 但是在这种情况下,如何使用经过训练的模型在PySpark中进行预测?
It doesn't seem possible to serialise keras models, so maybe just distribute the file and as a spark file? 似乎不可能序列化keras模型,所以也许只分发文件并作为spark文件? So inside your function (where you expect the model as input) you can read file from that path and create model inside it?
那么在函数内部(您希望模型作为输入),您可以从该路径读取文件并在其中创建模型吗?
path = SparkFiles.get("mode_file.h5")
model = load_model(path)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.