如何在HDP中安装scala库（Hortonworks数据平台）

Question

thanks in advance for the time you according reading this and sorry for my bad english.

I am trying to use Spark streaming for real time data processing. I have Spark installed in HDP (Hortonworks Data Platform) and for my process I need to install a scala library for JSONparsing. I read a lot of things on internet about that but it just was for a simple Spark Cluster not for solution like HDP and CDH, I tried to adapt the solution but I couldn't, I don't find any scala files to install it. Does anybody know a solution or a tips to help me ? Thank you

Answer 1

To load dependencies for Spark in Zeppelin you need to create a new cell and use the following:

%dep
// it's a good idea to do a reset first, but not required
z.reset()
// the following line will load directly from the Maven online repo
z.load("org.apache.spark:spark-streaming-karka_2.10:1.6.1")

Additional details on loading dependencies for Zeppelin can be found here: https://zeppelin.apache.org/docs/latest/interpreter/spark.html#3-dynamic-dependency-loading-via-dep-interpreter

One thin to not here, is that dependency loading must be the first cell you run on your notebook, it will give you an error message if it's not. To get around this click on the Interpreter tab and click restart on the Spark Interpreter, then go back to your notebook and run the cell with the %dep