writing json file using arrays with large data using java

Question

I am trying to form a json file to source an autocomplete controlled textbox.

The file will have millions of elements so I am trying to eliminate duplicates while saving on memory and time. For small amount the following code works yet since I am using an array, the execution gets really slow as the array gets larger.

int i = 0;
JSONObject obj = new JSONObject();     
JSONArray array = new JSONArray();

while (iter.hasNext()) {
    Map<String,String>forJson = new HashMap<String, String>();
    Statement stmt = iter.nextStatement();

    object = stmt.getObject();


    forJson.put("key", object.asResource().getLocalName());
    forJson.put("value", object.asResource().getURI());


    i++;
    System.out.println(i);
    if(!array.contains(forJson))
    {
        array.add(forJson);
    }
} 
obj.put("objects", array);

FileWriter file = new FileWriter("/homeDir/data.json");
file.write(obj.toJSONString());
file.flush();
file.close();

The array.contains control eliminates duplicates but it has a considerable negative effect on execution time.

The json file should have tokens like

[{"key": "exampleText1", "value": "exampleValue1"},
{"key": "exampleText2", "value": "exampleValue2"}]

Answer 1

Use a HashSet to contain the keys you have already added:

...
Set<String> usedKeys = new HashSet<String>();
while (iter.hasNext()) {
    Map<String,String>forJson = new HashMap<String, String>();
    Statement stmt = iter.nextStatement();

    object = stmt.getObject();

    String key = object.asResource().getLocalName();
    if(!usedKeys.contains(key)) {
        usedKeys.add(key);
        forJson.put("key", key);
        forJson.put("value", object.asResource().getURI());
        array.add(forJson);
    }

    i++;
    System.out.println(i);
} 

If you need to uniqueness check to include the value, you could append the two using a character separator that you know cannot exist in the keys. For example:

String key = object.asResource().getLocalName();
String value = object.asResource().getURI();
String unique = key + "|@|@|" + value;
if(!usedKeys.contains(unique)) {
    usedKeys.add(unique);
    forJson.put("key", key);
    forJson.put("value", value);
    array.add(forJson);
}