改變RDD的特定領域

Question

我是新手，我對轉換RDD的特定領域存有疑問。 我有一個如下文件：

2016-11-10T07:01:37|AAA|S16.12|MN-MN/AAA-329044|288364|2|3
2016-11-10T07:01:37|BBB|S16.12|MN-MN/AAA-329044/BBB-1|304660|0|0
2016-11-10T07:01:37|TSB|S16.12|MN-MN/AAA-329044/BBB-1/TSB-1|332164|NA|NA
2016-11-10T07:01:37|RX|S16.12|MN-MN/AAA-329044/BBB-1/TSB-1/RX-1|357181|0|1

我希望輸出如下：在第三個字段中，我要刪除所有用|分隔的字符和整數。

2016-11-10T07:01:37|AAA|16.12|329044|288364|2|3
2016-11-10T07:01:37|BBB|16.12|329044|1|304660|0|0
2016-11-10T07:01:37|TSB|16.12|329044|1|1|332164|NA|NA
2016-11-10T07:01:37|RX|16.12|329044|1|1|1|357181|0|1

我怎樣才能做到這一點。 我嘗試了以下代碼。

val inputRdd =sc.textFile("file:///home/arun/Desktop/inputcsv.txt");
val result =inputRdd.flatMap(line=>line.split("\\|")).collect;
def ghi(arr:Array[String]):Array[String]=
{
 var outlist=scala.collection.mutable.Buffer[String]();
 for( i <-0 to arr.length-1){
if(arr(i).matches("(.*)-(.*)")){
var io=arr(i);  var arru=scala.collection.mutable.Buffer[String](); 
if(io.contains("/"))
{
var ki=io.split("/");
for(st <-0 to ki.length-1 )
{
 var ion =ki(st).split("-");
 arru+=ion(1);
}  
var strui="";
for(in <-0 to arru.length-1)
{
strui=strui+arru(in)+"|";
}
outlist+=strui;
}
else
{           
var ion =arr(i).split("-");
outlist+=ion(1)+"|";
}
}
else
{
outlist+=arr(i);
}
}   
return outlist.toArray;
}
var output=ghi(result);
val finalrdd=sc.parallelize(out, 1);
finalrdd.collect().foreach(println);

請幫我。

Answer 1

我們需要做的是從該字段中提取數字並將其作為新條目添加到正在處理的Array中。

這樣的事情應該做：

// use data provided as sample 
val dataSample ="""2016-11-10T07:01:37|AAA|S16.12|MN-MN/AAA-329044|288364|2|3
2016-11-10T07:01:37|BBB|S16.12|MN-MN/AAA-329044/BBB-1|304660|0|0
2016-11-10T07:01:37|TSB|S16.12|MN-MN/AAA-329044/BBB-1/TSB-1|332164|NA|NA
2016-11-10T07:01:37|RX|S16.12|MN-MN/AAA-329044/BBB-1/TSB-1/RX-1|357181|0|1""".split('\n')

val data = sparkContext.parallelize(dataSample)

val records= data.map(line=> line.split("\\|"))

// this regex can find and extract the contiguous digits in a mixed string.
val numberExtractor = "\\d+".r.unanchored

// we replace field#3 with the results of the regex
val field3Exploded = records.map{arr => arr.take(3) ++ numberExtractor.findAllIn(arr.drop(3).head) ++ arr.drop(4)}

// Let's visualize the result
field3Exploded.collect.foreach(arr=> println(arr.mkString(",")))

2016-11-10T07:01:37,AAA,S16.12,329044,288364,2,3
2016-11-10T07:01:37,BBB,S16.12,329044,1,304660,0,0
2016-11-10T07:01:37,TSB,S16.12,329044,1,1,332164,NA,NA
2016-11-10T07:01:37,RX,S16.12,329044,1,1,1,357181,0,1