有没有一种方法可以强制等待Java进程结束在Scala或Spark中结束？

Question

In a Scala app, deployed through Spark, I have a code line which calls to a Java function executing native C++ code through a JNI. This call takes time, and if it is not the only one of his kind running, a resource usage conflict appears with a *** stack smashing detected ***: <unknown> terminated .

Here's the call, and it's scope :

[spark RDD].mapPartitionsWithIndex(f = (index: Int, it: Iterator[Row]) => {
  val sourceData: String = it.mkString()

  val result: List[List[String]] = new WrapperClass(sourceData, [misc parameters).getResult

  [wrinting result to a file]
}).take(1)

My WrapperClass.getResult , very simple, looks like this :

[java call related variables initialization]

UnitexJni.execUnitexTool("UnitexTool {InstallLingResourcePackage -p " + appliedPkg + " -x " + resDir + "} " + "{" + runScriptCmd + "} " + "{InstallLingResourcePackage -p " + appliedPkg + " -x " + resDir + " --uninstall}")

[retrieving, formatting and returning result]

The UnitexJni.execUnitexTool() is the java call.

So I would like to know if there is a way to force to wait until the end of this process before calling it over agin using Scala, Java or Spark functionality.

Answer 1

You could use sys.process._ You will pass the shell script to the below process function with the script path. Also, you need to handle the shell script to have a return code. For example, If 0 success else failed . Please take care of the ! at the end of the line. You could also check more details to run quick command lines from this tutorial

import scala.sys.process.Process
val externalShellScript = Process("sh", Seq(scriptPath)).!    
  if (externalShellScript != 0) {
    throw new Exception(
      "Error in executing external shell script from " + scriptPath)
  }

The Spark job will not continue unless this process finish. Below is simple shell script and the output.

touch test.txt
echo "any massage"

output in the console will be

any massage