如何在Scala Spark中反向映射和遍历键和值

Question

I am writing a Scala Spark script where I am storing (Hive partitions) HDFS paths as keys and their underlying file count as values. I want to write a for loop to check if the filepath exists (using key) and count is greater than 0. I have to reverse the whole map because I want to start checking for latest partition.

I couldn't find any method to reverse the map and loop through it. Here's my code:

val paths = collection.mutable.Map[String, Float]()
status.foreach( x => paths += (x.getPath.toString -> fs.getContentSummary(new Path(x.getPath.toString).getFileCount() )

This is what I am planning to do :

//Reverse the map to make last element as first element
var result = ""
for ((k,v) <- paths) {
if(!fs.exists(new Path(k)) && v < 1)
  continue
else
  result = k
  break
}

Answer 1

"Up Vote" if answer works for you.

You need to use ListMap and then you sort map based on key ( basically you can reverse the map and perform further operations ).

import scala.collection.immutable.ListMap

val paths_sortBy_key_asc = ListMap(paths.toSeq.sortBy(_._1) < _. 1: *)

or 

val paths_sortBy_key_dsc = ListMap(paths.toSeq.sortBy(_._1) > _. 1: *)

Hope this helps !!

Answer 2

There are multiple options.

1. Use TreeMap to construct data structure. Its is sorted by keys.
2. Convert it to a seq or list and sort it.

All these options are outlined here with an example.

refer to sort map link