[英]Multiple file storage in HDFS using Apache Spark
我正在做一個涉及使用HDFS進行存儲和使用Apache Spark進行計算的項目。 我在HDFS中有一個目錄,該目錄中有多個深度相同的文本文件。我想使用Spark處理所有這些文件並將其對應的結果存儲回HDFS,每個輸入文件有1個輸出文件。
例如-假設我在同一深度下有一個包含1000個文本文件的目錄。 我正在使用通配符讀取所有這些文件
sc.wholeTextFiles(hdfs://localhost:9000/home/akshat/files/*.txt)
然后我使用Spark處理它們並獲得相應的RDD並通過使用保存
result.saveAsTextFile("hdfs://localhost:9000/home/akshat/final")
但這給了我所有輸入文件的結果在一個文件中,我想獲取每個文件,分別處理它們,並分別存儲它們的輸出。
實現該目標的下一個方法是什么?
提前致謝!
您可以通過使用WholeTextFiles()來做到這一點, 注意:以下方法一步一步地處理文件。
val data = sc.wholeTextFiles("hdfs://master:port/vijay/mywordcount/")
val files = data.map { case (filename, content) => filename}
def doSomething(file: String) = {
println (file);
// your logic of processing a single file comes here
val logData = sc.textFile(file);
val numAs = logData.filter(line => line.contains("a")).count();
println("Lines with a: %s".format(numAs));
// save rdd of single file processed data to hdfs comes here
}
files.collect.foreach( filename => {
doSomething(filename)
})
哪里:
更新:多個輸出文件
/* SimpleApp.scala */
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
/* hadoop */
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.NullWritable
import org.apache.hadoop.mapred.lib.MultipleTextOutputFormat
/* java */
import java.io.Serializable;
import org.apache.log4j.Logger
import org.apache.log4j.Level
/* Custom TextOutput Format */
class RDDMultipleTextOutputFormat extends MultipleTextOutputFormat[Any, Any] {
override def generateActualKey(key: Any, value: Any): Any =
NullWritable.get()
override def generateFileNameForKeyValue(key: Any, value: Any, name: String): String =
return key.asInstanceOf[String] +"-"+ name; // for output hdfs://Ouptut_dir/inputFilename-part-****
//return key.asInstanceOf[String] +"/"+ name; // for output hdfs://Ouptut_dir/inputFilename/part-**** [inputFilename - as directory of its partFiles ]
}
/* Spark Context */
object Spark {
val sc = new SparkContext(new SparkConf().setAppName("test").setMaster("local[*]"))
}
/* WordCount Processing */
object Process extends Serializable{
def apply(filename: String): org.apache.spark.rdd.RDD[(String, String)]= {
println("i am called.....")
val simple_path = filename.split('/').last;
val lines = Spark.sc.textFile(filename);
val counts = lines.flatMap(line => line.split(" ")).map(word => (word, 1)).reduceByKey(_ + _); //(word,count)
val fname_word_counts = counts.map( x => (simple_path,x._1+"\t"+ x._2)); // (filename,word\tcount)
fname_word_counts
}
}
object SimpleApp {
def main(args: Array[String]) {
//Logger.getLogger("org").setLevel(Level.OFF)
//Logger.getLogger("akka").setLevel(Level.OFF)
// input ans output paths
val INPUT_PATH = "hdfs://master:8020/vijay/mywordcount/"
val OUTPUT_PATH = "hdfs://master:8020/vijay/mywordcount/output/"
// context
val context = Spark.sc
val data = context.wholeTextFiles(INPUT_PATH)
// final output RDD
var output : org.apache.spark.rdd.RDD[(String, String)] = context.emptyRDD
// files to process
val files = data.map { case (filename, content) => filename}
// Apply wordcount Processing on each File received in wholeTextFiles.
files.collect.foreach( filename => {
output = output.union(Process(filename));
})
//output.saveAsTextFile(OUTPUT_PATH); // this will save output as (filename,word\tcount)
output.saveAsHadoopFile(OUTPUT_PATH, classOf[String], classOf[String],classOf[RDDMultipleTextOutputFormat]) // custom output Format.
//close context
context.stop();
}
}
環境:
樣本輸出:
[ramisetty@node-1 stack]$ hadoop fs -ls /vijay/mywordcount/output
Found 5 items
-rw-r--r-- 3 ramisetty supergroup 0 2015-06-09 03:49 /vijay/mywordcount/output/_SUCCESS
-rw-r--r-- 3 ramisetty supergroup 40 2015-06-09 03:49 /vijay/mywordcount/output/file1.txt-part-00000
-rw-r--r-- 3 ramisetty supergroup 8 2015-06-09 03:49 /vijay/mywordcount/output/file1.txt-part-00001
-rw-r--r-- 3 ramisetty supergroup 44 2015-06-09 03:49 /vijay/mywordcount/output/file2.txt-part-00002
-rw-r--r-- 3 ramisetty supergroup 8 2015-06-09 03:49 /vijay/mywordcount/output/file2.txt-part-00003
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.