如何针对每个RDD Spark流

Question

I have one CSV file queries.txt and I am reading the file like this:

JavaRDD<String> distFile = sc.textFile("queries.txt");

Schema of queries.txt file is: Uniq_Id,,,...some numeric values in csv...

I need for each line - create a HashMap, whose key is first column of the queries.txt file(Uniq_Id) and value is other columns in file to HashMap.

example. (This is not real and not a working example, I just want to convey the essence)

HashMap totalMap = new HashMap<Integer, NumericValues>();

for(int i=0;i<distFile.size();i++)
{
   String line = distFile[i].getColumns();
   for(int y=0;y<line.size();y++)
   {
      totalMap.put(line.getFirstColumn,line.getRemainingColumns);
   }
}

Here NumericValues is my custom class which will have the variables mapping to the columns in the file.

Any other suggestions will be helpful.

Answer 1

I guess this is what you are looking for, but this example doesn't parses the CSV line itself.

  JavaRDD<String> distFile = sc.textFile("queries.txt");
  HashMap totalMap = new HashMap<Integer, NumericValues>();
  distFile.foreach(new VoidFunction<String>(){ 
          public void call(String line) {
              totalMap.put(yourCSVParser(line)); //this is dummy function call 
    }});