Batch insert to multiple tables with foreign keys using JDBC

Question

I am writing a method that receives a tree structure and inserts it to database, where every branch is another database table. For example:

tree = {
    "_id": "111",
    "field1": "",
    "field2": "",
    "field3": {
        "_id": "333",
        "_parent_id": "111",
        "field3_1": "",
        "field3_2": "",
        "field3_3": [
            {
                "_id": "1",
                "_parent_id": "333",
                "field3_3_1": ""
            },
            {
                "_id": "2",
                "_parent_id": "333",
                "field3_3_1": ""
            },
            {
                "_id": "3",
                "_parent_id": "333",
                "field3_3_1": ""
            }
        ]
    }
}

Should correspond to 3 tables: tree, field3 and field3_3 - that have relations on _id and _parent_id. Every branch can be single value, table or object, basically the tree is dynamic structure that corresponds to some data structure in PostgreSQL database. I am using jdbc to insert the data to database. What is the best way to insert such data when it is possible that user will supply big trees with multiple branches or multiple branch values? I was thinking about statement like:

                 with first_insert as(
                 insert into sample(firstname, lastname)
                 values('fai55', 'shaggk')
                 RETURNING id
                 ),

                 second_insert as(
                 insert into sample1(id, adddetails)
                 values
                 ((select id from first_insert), 'ss')
                 RETURNING user_id
                 )

But the problem will hit if the inner insert should work on big batch records (field3_3 would contain 100k records for every data row).

I have already implemented method that accepts table and inserts is on single DB level, using PreparedStatement and executeBatch() and it works great. Anything like this method, available for trees?

Answer 1

If the tree is dynamic structure I think you will have to parse the json you recieve into something you can work in Java.

Such thing can be achieved using and JSONObject . Suppose you got incoming json serialized in String data . Then you can use:

JSONObject dataJson = new JSONObject(data);

Now you can work with this tree structure in Java. You can serialize it back using dataJson.toString() or create a custom serializer.

If the problem is the insert size, you can loop over dataJson and split it into reasonable chunks. For example you can use something like this

        JSONArray jsonArr = dataJson.getJSONArray("field3");
        JSONArray dataAccumJsonArr = new JSONArray();
        for (int i = 0; i < jsonArr.length(); i++) {
            dataAccumJsonArr.put(jsonArr.get(i));
            if (dataAccum.length() > 999) {
                doInsert(dataAccumJsonArr);
                dataAccumJsonArr = new JSONArray();
            }
        }
        doInsert(dataAccumJsonArr);

Answer 2

I have solved the problem by following: I take elements from one level of the tree and and insert all of its non object values in preparedStatement batch (they are in one table), returning the inserted record ids. Then I take children of each of the inserted element, assign the correct parent_id and batch insert them to another table, and another. Works great!