Adding h5 files in a zip to use with PySpark

Question

I am using PySpark 1.6.1 for my spark application. I have additional modules which I am loading using the argument --py-files . I also have a h5 file which I need to access from one of the modules for initializing the ApolloNet.

Is there any way I could access those files from the modules if I put them in the same archive? I tried this approach but it was throwing an error because the files are not there in every worker. I can think of copying the file to each of the workers but I want to know if there are better ways to do it?

Answer 1

发现您可以通过使用spark-submit中的--files参数将其他文件添加到所有工作程序。