How to parse json in Spring Boot and write to a repository

Question

I am trying to parse a JSON API response of historical time series data with potentially thousands of line. The response is in the following format:

{
    "name": "AAPL",
    "history": {
        "2019-03-05": {
            "open": "175.94",
            "close": "175.53",
            "high": "176.00",
            "low": "174.54",
            "volume": "19163899"
        },
        "2019-03-04": {
            "open": "175.69",
            "close": "175.85",
            "high": "177.75",
            "low": "173.97",
            "volume": "27436203"
        }
    }
} 

I would like to write the response to a Spring repository. I have a simple code to do this and a section is shown below:

RestTemplate restTemplate = new RestTemplate();
JsonParser jsonParser = new JsonParser();
JsonObject jsonObject = (JsonObject) jsonParser.parse(result);
JsonElement jsonElement = jsonObject.get("history");
Set<Map.Entry<String, JsonElement>> entrySet = jsonElement.getAsJsonObject().entrySet();

for (Map.Entry<String, JsonElement> entry : entrySet) {
    StockHistory stockHistory = new StockHistory();
    stockHistory.setSymbol(stk);
    // .... Other code
}

I set the object properties as per JSON response, add the object to a list, and finally save the list to a repository. This process is very slow presumably because I am creating a new StockHistory object for every element in the JSON return. I was wondering if there is a better way of doing it.

Answer 1

As you cannot modify the JSON structure. I would like to add the following code that can parse the JSON that you provided in a simple class called Repo . In order to do that you need to add the library from here that I have used.

Now you need to add the following classes in your code.

public class Repo {
    public String name;
    public ArrayList<History> histories;

    public Repo() {
        histories = new ArrayList<History>();
    }
}

public class History {
    public String date;
    public HistoryElements elements;
}

public class HistoryElements {
    public String volume;
    public String high;
    public String low;
    public String close;
    public String open;
}

Hence I have written a RepoParser and tested it with your JSON String and it parses the JSON into the classes.

import com.oracle.javafx.jmx.json.JSONException;
import org.json.JSONObject;

import java.text.ParseException;
import java.util.ArrayList;
import java.util.Iterator;

public class RepoParser {
    public static Repo parseRepo(String jsonString) throws ParseException, JSONException {

        JSONObject jsonObject = new JSONObject(jsonString);
        Iterator<?> iterator = jsonObject.keys();
        Repo repo = new Repo();

        while (iterator.hasNext()) {
            String obj = iterator.next().toString();

            if (obj.equals("name")) repo.name = obj;
            else repo.histories = parseHistory((JSONObject) jsonObject.get(obj));
        }

        return repo;
    }

    public static ArrayList<History> parseHistory(JSONObject jsonObject) throws ParseException, JSONException {
        Iterator<?> iterator = jsonObject.keys();
        ArrayList<History> historyList = new ArrayList<>();

        while (iterator.hasNext()) {
            String obj = iterator.next().toString();
            History history = new History();
            history.date = obj;
            history.elements = parseHistoryElement((JSONObject) jsonObject.get(obj));

            historyList.add(history);
        }

        return historyList;
    }

    public static HistoryElements parseHistoryElement(JSONObject jsonObject) throws ParseException, JSONException {
        Iterator<?> iterator = jsonObject.keys();
        HistoryElements historyElements = new HistoryElements();

        while (iterator.hasNext()) {
            String obj = iterator.next().toString();

            if (obj.equals("open")) historyElements.open = jsonObject.getString("open");
            if (obj.equals("close")) historyElements.close = jsonObject.getString("close");
            if (obj.equals("high")) historyElements.high = jsonObject.getString("high");
            if (obj.equals("low")) historyElements.low = jsonObject.getString("low");
            if (obj.equals("volume")) historyElements.volume = jsonObject.getString("volume");
        }

        return historyElements;
    }
}

Just use the RepoParser class like the following.

try {
    Repo repo = RepoParser.parseRepo(jsonString);
} catch (Exception e) {
    e.printStackTrace();
}

I have created a Gist as well for convenience.

Update

You might consider adding all the Repo in a list and then save all of them into your database at once using the save method of the repository.

Hence the code should be something like the following.

try {
    while(there is no repo left for parsing) {
        Repo repo = RepoParser.parseRepo(jsonString);
        repoList.add(repo)
    }

    yourRepository.save(repoList); // Save all values at once

} catch (Exception e) {
    e.printStackTrace();
}

Hope that helps!

Answer 2

After some research, I found that the problem was with hibernate. As far as I understand it, a useful feature of hibernate is that it caches objects, but this causes a problem when a large number of objects are created for insertion. The issue can be resolved by batch processing using the spring.jpa.properties.hibernate.jdbc.batch_size property and using a sequence generator in the entity class. Now saving the list of many thousands of lines is much faster.