Reading one file as Map(K,V) and pass V as keys while reading the second file as Map

Question

I have two files. One is a text file and another one is CSV. I want to read the text file as Map(keys, values) and pass these values from the first file as key in Map when I read the second file (CSV file).

I am able to read the first file and get Map(key, value). From this Map, I have extracted the values and passed these values as keys in the second file but didn't get the desired result.

first file - text file

sdp:field(0)
meterNumber:field(1)
date:field(2)
time:field(3)
value:field(4),field(5),field(6),field(7),field(8),field(9),
field(10),field(11),field(12),field(13),field(14),
field(15),field(16),field(17)

second file - csv file

SDP,METERNO,READINGDATE,TIME,Reset Count.,Kilowatt-Hour Last Reset .,Kilowatt-Hour Rate A Last Reset.,Kilowatt-Hour Rate B Last Reset.,Kilowatt-Hour Rate C Last Reset.,Max Kilowatt Rate A Last Reset.,Max Kilowatt Rate B Last Reset.,Max Kilowatt Rate C Last Reset.,Accumulate Kilowatt Rate A Current.,Accumulate Kilowatt Rate B Current.,Accumulate Kilowatt Rate C Current.,Total Kilovar-Hour Last Reset.,Max Kilovar Last Reset.,Accumulate Kilovar Last Reset.
9000000001,500001,02-09-2018,00:00:00,2,48.958,8.319333333,24.31933333,16.31933333,6,24,15,10,9,6,48.958,41,40

this is what I have done to read the first file.

val lines = scala.io.Source.fromFile("D:\\JSON_READER\\dailymapping.txt", "UTF8")
        .getLines
        .map(line=>line.split(":"))
        .map(fields => (fields(0),fields(1))).toMap;        
  val sdp = lines.get("sdp").get;
  val meterNumber = lines.get("meterNumber").get;
  val date = lines.get("date").get;
  val time = lines.get("time").get;
  val values = lines.get("value").get;

now I can see sdp has field(0), meterNumber has field(1), date has field(2), time has field(3) and values has field(4) .. to field(17).

Second file which I m reading using below code

val keyValuePairs = scala.io.Source.fromFile("D:\\JSON_READER\\Daily.csv")
       .getLines.drop(1).map(_.stripLineEnd.split(",", -1))
       .map{field => ((field(0),field(1),field(2),field(3)) -> (field(4),field(5)))}.toList

  val map = Map(keyValuePairs : _*)
  System.out.println(map);

above code giving me the following output which is desired output.

Map((9000000001,500001,02-09-2018,00:00:00) -> (2,48.958))

But I want to replace field(0), field(1), field(2), field(3) with sdp, meterNumber, date, time in the above code. So, I don't have to mention keys when I read the second file, keys will come from the first file.

I tried to replace but I got below output which is not desired output.

Map((field(0),field(1),field(2),field(3)) -> (,))

Can somebody please guide me on how can I achieve the desired output.

Answer 1

This might get you close to what you're after. The first Map is used to lookup the correct index into the CSV data.

val fieldRE = raw"field\((\d+)\)".r

val idx = io.Source
            .fromFile(<txt_file>, "UTF8")
            .getLines
            .map(_.split(":"))
            .flatMap(fields => fieldRE.replaceAllIn(fields(1), _.group(1))
                                      .split(",")
                                      .map(fields(0) -> _.toInt))
            .toMap

val resMap = io.Source
               .fromFile(<csv_file>)
               .getLines
               .drop(1)
               .map(_.stripLineEnd.split(",", -1))
               .map{ fld =>
  (fld(idx("sdp")),fld(idx("meterNumber")),fld(idx("date")),fld(idx("time"))) ->
  (fld(4),fld(5))  }
               .toMap

//resMap: Map((9000000001,500001,02-09-2018,00:00:00) -> (2,48.958))

---

UPDATE

Changing the Map of ( String identifiers -> Int index values) into a Map of ( String identifiers -> collection of Int index values) can be done. I'm not sure what that buys you, but it's doable.

val fieldRE = raw"field\((\d+)\)".r

val idx = io.Source
            .fromFile(<txt_file>, "UTF8")
            .getLines
            .map(_.split(":"))
            .flatMap(fields => fieldRE.replaceAllIn(fields(1), _.group(1))
                                      .split(",")
                                      .map(fields(0) -> _.toInt))
            .foldLeft(Map[String,Seq[Int]]()){ case (m,(k,v)) =>
               m + (k -> (m.getOrElse(k,Seq()) :+ v))
            }

val resMap = io.Source
               .fromFile(<csv_file>)
               .getLines
               .drop(1)
               .map(_.stripLineEnd.split(",", -1))
               .map{fld => (fld(idx("sdp").head)
                           ,fld(idx("meterNumber").head)
                           ,fld(idx("date").head)
                           ,fld(idx("time").head)) -> (fld(4),fld(5))}
               .toMap