与bash脚本并行运行多个Scala主类

Question

I have an application with Kafka data producer and Spark consumer, where KafkaProducer object extends App and SparkConsumer defined the main method. I want to create a bash script so I could set which class to run - producer or consumer and run them in parallel. I have managed to create such a script, but sbt takes a while to load and I need to restart producer multiple times which takes much longer than just running the same class in IDE. Where can I move sbt command definition or which approach can I choose to decrease the time needed to run an application?

PS I run both consumer and producer separately in different terminals.

Here is how my bash script looks like:

#!/usr/bin/env bash
if [ "$1" = "consumer" ]
then
    sbt "runMain consumer.SparkConsumer $2 $3 $4"
elif [ "$1" = "producer" ]
then
    sbt "runMain producer.KafkaProducer $5 $3 $6 $7"
else
    echo "Wrong parameter. It should be consumer or producer"
fi

Answer 1

You have several options here:

Maybe you don't know it, but your sbt compiles your Scala code into java bytecode (a .jar) file and then runs it using java . So you could do that directly yourself:

• run sbt package to compile your code
• run your code with java -cp "target/scala-<SCALA_VERSION>/<PROJECT_NAME>-<PROJECT_VERSION>.jar" your.main.class.Name

( <SCALA_VERSION> , <PROJECT_NAME> , <PROJECT_VERSION> and your.main.class.Name have to be replaced with your own values)

This should allow you to start your command faster, as sbt takes a while to start. You will still have the overhead of the JVM starting time however, which leads me to the second solution:

If you really need to start your commands quickly, then I suggest that you modify your scala program to be able to accept an arbitrary number of actions from the command line (or by reading a file) launch the commands in parallel directly in your scala code. This is as easy as this: Seq(1, 2, 3, 4).par.foreach{println} The .par will create a ParSeq which is a sequence that can run in parallel. You can even configure the degree of parallelism, but that is another question.

Third option: you could have a look at https://github.com/facebook/nailgun (or any similar project ) and use it to reduce your JVM starting time overhead.