支持Map的IDE减少Scala中的程序

Question

Hi can anyone suggest the suitable IDE for writing mapreduce programs in Scala? Please provide the sample code for WordCount Program in Scala using mapreduce.

Answer 1

If you are talking about distributed computations so the main player in this area in scala world is Spark .

Words count example is coming with documentation :

val textFile = spark.textFile("hdfs://...")
val counts = textFile.flatMap(line => line.split(" "))
                 .map(word => (word, 1))
                 .reduceByKey(_ + _)
counts.saveAsTextFile("hdfs://...")

You can run this code in interactive shell or in your program. Spark context could be created this way:

val conf = new SparkConf().setAppName("Simple Application")
val spark = new SparkContext(conf)

There are also many ready to run examples on Github

Answer 2

You can use eclipse as Spark application is essentially scala code.
Any ide will only help in developing (writing the code) and not in debugging since this application is submitted to Hadoop cluster and executed parallely in all the nodes.