使用Apache Spark指定特定计算机

Question

I'm totally new to Spark and Hadoop-type stuff in general, so forgive me if this is a painfully basic question. I'm trying to design a system that will make use of a cluster of some number of machines to do the first tasks in a series of tasks. The follow-up tasks, which run on the RDDs that the first tasks generate, must all be done on the same machine. This could be any machine from the cluster as long as it's always that machine for the duration of the program run.

How do I make sure that happens? Can I reserve a single machine in the cluster and always run the follow-up tasks on that machine? If so, how does that look in Java? If not, is there some other way to accomplish this?

Answer 1

In general, no. Spark, like Hadoop, is designed to distribute tasks more or less arbitrarily, over the available nodes, and assumes all available nodes are equivalent for its purposes. None of them will be treated specially.

If you don't want the second half of the process to run in a (more or less) massively parallel fashion, you probably don't want to use a parallel processing framework for that half of the job. Maybe you should write all the data from the parallel calculations to disk somewhere, and then run the second half of the job on the data, not as Spark RDD transformations, but just as normal Scala code which reads the files and processes them. It's hard to say.

Why do all the "follow-up tasks" need to run in one particular place? If you can explain more about this need, maybe someone can make good suggestions for you.