Java: how to perform batch insert with identity column using java jdbc for Sql Server

Question

I have a csv file which I need to write to a Sql Server table using SQLServerBulkCopy . I am using SQLServerBulkCSVFileRecord to load data from the file.

The target table has the following structure:

create table TEST
(
    ID int identity,
    FIELD_1 int,
    FIELD_2 varchar(20)
)

The csv file has the following structure:

4279895;AA00000002D
4279895;AA00000002D
4279895;AA00000002D
4279896;AA00000003E
4279896;AA00000003E
4279896;AA00000003E

As you can see the ID (identity) column is not present in the csv, I need the database to automatically add the identity value on insert. My problem is that the bulk insert does not work as long as the table has the identity column, I got the following error:

com.microsoft.sqlserver.jdbc.SQLServerException: Source and destination schemas do not match.
    at com.microsoft.sqlserver.jdbc.SQLServerBulkCopy.validateColumnMappings(SQLServerBulkCopy.java:1749)
    at com.microsoft.sqlserver.jdbc.SQLServerBulkCopy.writeToServer(SQLServerBulkCopy.java:1579)
    at com.microsoft.sqlserver.jdbc.SQLServerBulkCopy.writeToServer(SQLServerBulkCopy.java:606)

This is the relevant code:

try (
        Connection targetConnection = DriverManager.getConnection(Configuration.TARGET_CONNECTION_URL);
        SQLServerBulkCopy bulkCopy = new SQLServerBulkCopy(targetConnection);
        SQLServerBulkCSVFileRecord fileRecord = new SQLServerBulkCSVFileRecord(csvPath, Charsets.UTF_8.toString(), ";", false);
) {

    SQLServerBulkCopyOptions copyOptions = new SQLServerBulkCopyOptions();
    copyOptions.setKeepIdentity(false);

    bulkCopy.setBulkCopyOptions(copyOptions);

    fileRecord.addColumnMetadata(1, null, java.sql.Types.INTEGER, 0, 0); // FIELD_1 int    
    fileRecord.addColumnMetadata(2, null, java.sql.Types.VARCHAR, 20, 0); // FIELD_2 varchar(20)

    bulkCopy.setDestinationTableName("TEST");                
    bulkCopy.writeToServer(fileRecord);

}

catch (Exception e) {

    // [...]

}

The bulk insert ends succesfully if I remove the identity column from the table. Which is the correct to perform an identity bulk-insert using java jdbc for Sql Server?

Answer 1

I think you don't need to set this option copyOptions.setKeepIdentity(false);

Try after removing this line. You can refer to this post as well SqlBulkCopy Insert with Identity Column

Answer 2

If you have a leading column with blank values Helen the identity will be generated on insert. Depending on the settings it might generate new identity even if the first column is not blank.

So either add an extra column or use another (staging) table.

BTW, if you have a really big table the command-line bcp utility is the fastest. From experience up to 5 times faster compared to Jdbc batch insert.