如何使用 scala 在 Apache Spark 中运行 shell 文件

Question

I need to execute a shell file at the end of my code in spark using scala. I used count and groupby functions in my code. I should mention that, my code works perfectly without the last line of code.

(import sys.process._
/////////////////////////linux commands
val procresult="./Users/saeedtkh/Desktop/seprator.sh".!!)

could you please help me how to fix it.

Answer 1

You must use sys.process._ package from Scala SDK and use DSL with ! :

import sys.process._
val result = "ls -al".!

Or make same with scala.sys.process.Process :

import scala.sys.process._
Process("cat data.txt")!

Answer 2

You almost there. You need to add .!! to the end of the command.For example:

import sys.process._
val a = Seq("sh", "-c", " cal ").!!
println(a)

The reason i used Seq, is that you have more control about the parameters that you pass. see this post: How to execute shell builtin from Scala