在Spark Pipeline中部署Tensorflow / Keras模型

Question

I have trained several RNN+biLSTM models that I want to deploy in a pipeline consisting of pyspark pipeline steps. spark-deep-learning seems to be a stale project that only accommodates work with image data. Are there any best practices today for loading tensorflow / keras models (and their associated vector embeddings) into pyspark pipelines?

Answer 1

If you want to deploy a tensorflow model into Spark, you should take a look at Deeplearning4J . It comes with some Importers, where you can read keras and TensorFlow models. Be aware, that not every layer is supported.

Besides spark-deep-learning there is tensorframe , i never used it , so I don´t know how good it is.

In general I would suggest to use tensorflow directly via Distributed Tensorflow and not using all these wrappers.