Optimizing a HashMap with iteration

Question

I have a ResultSet containing 300K records and I am doing the following to iterate it (and other actions once collected). This process takes around 2 minutes to complete. Is there any way to optimize it?

Map<String,Map<String,String>> internalMap  = new HashMap<String,Map<String,String>>(); 

while (resultSet.next()) {
    final Integer val1 = resultSet.getInt("val1");
    final String val2 = resultSet.getString("val2");
    final String val3 = resultSet.getString("val3");
    final String val4 = resultSet.getString("val4");
    final String type = resultSet.getString("type");
    final String id = resultSet.getString("id");

    addIntern(internalMap,val2,val1,val3,val4,type,id);
}

And the addIntern method referenced above

private static void addIntern(Map<String,Map<String,String>> internalMap, String val2, Integer val1,
    String val3,String val4,String type,String id) {
    String key = id+"##"+val4;
    if (internalMap.get(key) == null) {
        internalMap.put(key, new HashMap<String,String>());
    }
    internalMap.get(key).put("val3", val3);
    internalMap.get(key).put("val2", val2);


    if("create".equals(type)){
        internalMap.get(key).put("create", val1.toString());
    }
    if("update".equals(type)){
        internalMap.get(key).put("update", val1.toString());
    }
    if("delete".equals(type)){
        internalMap.get(key).put("delete", val1.toString());
    }
}

Answer 1

Tune the fetch size from your resultSet ;

resultSet.setFetchSize(100);

And you could certainly simplify your add method (each get is an O(1) call, but they add up), like

private static void addIntern(Map<String, Map<String, String>> internalMap, String val2, Integer val1, String val3,
        String val4, String type, String id) {
    String key = id + "##" + val4;
    Map<String, String> kMap;
    if (internalMap.containsKey(key)) {
        kMap = internalMap.get(key);
    } else {
        kMap = new HashMap<>();
        internalMap.put(key, kMap);
    }
    kMap.put("val3", val3);
    kMap.put("val2", val2);
    if ("create".equals(type) || "update".equals(type) || "delete".equals(type)) {
        kMap.put(type, val1.toString());
    }
}

Answer 2

You won't be able to optimize much since you can't reduce number of iterations.

But one change I can see which can be done

 if("create".equals(type)){
    internalMap.get(key).put("create", val1.toString());
 }
 if("update".equals(type)){
    internalMap.get(key).put("update", val1.toString());
 }
 if("delete".equals(type)){
    internalMap.get(key).put("delete", val1.toString());
 }

Above can be written as below also

internalMap.get(key).put(type, val1.toString());

This will remove 300k or more if checks.

This will work in a case when your type can contain only create/update/delete values. If it has more you can put a single if to check if it is equal to any one of 3.

Answer 3

Try this. (Java 8)

private static void addIntern(Map<String,Map<String,String>> internalMap,
    String val2, Integer val1,
    String val3,String val4,String type,String id) {
    String key = id+"##"+val4;
    Map<String, String> valueMap = internalMap.computeIfAbsent(key, k -> new HashMap<>());
    valueMap.put("val3", val3);
    valueMap.put("val2", val2);
    if("create".equals(type) || "update".equals(type) || "delete".equals(type))
        valueMap.put(type, val1.toString());
}

Answer 4

Setting the map initial capacity may help if you have sufficient heap space available:

Map<String,Map<String,String>> internalMap  = new HashMap<String,Map<String,String>>(300000*2); 

By using the default initial capacity, internalMap must be rehashed many times since you are adding 300000 items.