使用 JDBC 在 Postgres 中插入具有复杂数据类型的大量数据的有效方法是什么

Question

We have a requirement to insert huge records in to postgres. In sqlServer we have used TableTypes for this. In postgres we have to do in something similar and i have done something like below.

I have a table MyuserDefinedTable with columns like

• id uuid
• date timestamp
• value integer
• name String

I have created a Function in postgres like,

CREATE or replace function myuserDefinedTableupdatefunction(
    _table_name regclass,
    _column_names text,
    _insert_arr MyuserDefinedTable[]) returns text as
$func$
BEGIN
    EXECUTE format('INSERT INTO %s(%s) SELECT * FROM unnest($1)',_table_name , _column_Names)   
        USING _insert_arr;          
    RETURN 'Saved'; 
EXCEPTION WHEN others THEN
    RETURN 'Failure';
END
$func$ LANGUAGE plpgsql;

Its working fine if i call the function, by passing params directly, like below,

Select myuserDefinedTableupdatefunction ('MyuserDefinedTable','ID,date,value,NAME', ARRAY[ROW('a1563404-0d1d-11ea-b563-3ca82a1c6940', '2019-11-21 00:00:00.0,100.00','3','hello'),ROW('b5563404-7d1d-11ea-b563-3ca82a1c6940', '2019-11-22 00:00:00.0,100.00','4','hii')]::MyuserDefinedTable[])

But its not executing (throwing error like, 'missed left Paranthesis' even though i ensured everything is fine syntactically) if i try to insert values dynamically through JDBC like below,

String query=Select myuserDefinedTableupdatefunction ('MyuserDefinedTable', 'ID,date,value,NAME', ARRAY[?]::MyuserDefinedTable[])

Object[] arr = objectList.toArray(new Object[0]);  // Object List has list of params appended like ( [0]: a1563404-0d1d-11ea-b563-3ca82a1c6940,2019-11-21 00:00:00.0,100.0,3,hello

// [1]: b1563404-0d1d-11ea-b563-3ca82a1c6940,2019-11-22 00:00:00.0,100.0,4,hi)
Array array = getSession().getConnection().createArrayOf("MyuserDefinedTable", arr);
ps.setArray(1, array);

Can anyone help me with the above?

Also, the execution time of inserting huge records is taking almost double than normal insert statements.

Is there any effective way of inserting huge records in to Postgres DB using an array with a complex datatype?

Edit : With some suggestions, I also tried using CopyManager like,

    String sql = "COPY MyuserDefinedTable FROM stdin CSV HEADER DELIMITER ','";
        BaseConnection pgcon = (BaseConnection)getSession().getConnection();
        CopyManager mgr = new CopyManager(pgcon);
        try {
            Reader in = new BufferedReader(new FileReader(new File("C:/Testing/data.csv"))); // The file is my local system
            long rowsaffected  = mgr.copyIn(sql, in);
            System.out.print(rowsaffected);
        }

But the above code returns 0 rows effected.

Any suggestions please.

Answer 1

Thanks All

The below solution worked for me, I tried using COPY function by passing Byte Arrays instead of getting data from files (As we are using Cloud, I came to know that file permissions will be a problem in AWS).

https://forums.aws.amazon.com/thread.jspa?threadID=141798

        List<String> valueList=myData();

        //Converting ArrayList to ByteArray
        try(ByteArrayOutputStream baos = new ByteArrayOutputStream()){
            DataOutputStream out = new DataOutputStream(baos);
            for (String element : valueList) {
                out.writeBytes(element);
            }       
            bytes = baos.toByteArray();
        }
        if(bytes!=null) {
            try(ByteArrayInputStream input = new ByteArrayInputStream(bytes)){

                // Copying records into table using COPY..FROM and COPYMANAGER in Postgres
                String sql = "COPY "+tableName+" FROM stdin delimiter ','  NULL AS 'null' ";
                BaseConnection pgcon = (BaseConnection)getSession().getConnection();
                CopyManager mgr = new CopyManager(pgcon);                   
                long rowsEffected=mgr.copyIn(sql,input);                                        
            }
        }

This seems faster than the way that i posted in the question above.