使用依赖项构建Apache Spark Mllib

Question

I don't want to run spark in a cluster. The only reason for using spark is to make use of the MLlib. In a nutshell, I need to use MLlib jar in my application with bare minimal dependencies. Currently, my spark assembly jar is around 125 MB. Is there any way to minimize it?

Answer 1

Depending on how an application gonna be used you can mark dependencies as provided, that will reduce the size of your jar, so deployments will be faster.

Also, you check if maven assembly also included Scala stdlib in jar ( sbt assembly by default includes Scala stdlib)