如何使我的 function 用于我的数据库的测试数据更快？

Question

I want to create testdata and have written a function for storing the products, my product generators generate in my database.

The plan is to create about 10,000,000 products or more for testing purposes.

I want to check every time before I insert a product, if the same product name exists.

If it does, the product isn't stored in the database. I know that the performance issue is the checking if the products exist, which takes longer and longer the more products are in the database. But there is no other way, I know, how I can improve this issue. I may use indexes, but I don't know how to in this scenario. If you have other ideas how to improve performance please feel free to comment your ideas.

tldr: I want to create testdata but it does take too long because it is checking if the products already exist. Want to improve performance.

Here is my code:

   public String insertProdukt(String name, Double preis, Integer kat_id) throws SQLException, ClassNotFoundException {
    Connection connection = ConnectionUtils.createNewConnection();
    // does the product exist?
    Statement statement = connection.createStatement();
    ResultSet resultSet = statement.executeQuery("select * from pro_produkte where pro_name=\"" + name + "\" AND pro_preis=\"" + preis + "\" AND pro_kat_id=\"" + kat_id + "\"");

    if (resultSet.next()) {
        //it does exist
        System.out.println("Produkt: " + resultSet.getString("pro_name") + " existiert bereits");
    } else {
        //it dosen't -> insert into database
        String sql = "Insert INTO pro_produkte (pro_name, pro_preis, pro_kat_id)"
                + "VALUES (\"" + name + "\", \"" + preis + "\", \"" + kat_id + "\")";
        statement.executeUpdate(sql);
        System.out.println("Produkt: " + name + " erstellt");

    }
    resultSet.close();
    statement.close();
    connection.close();
    return null;
}

Thanks!

Answer 1

Instead of simply INSERT... , use

INSERT IGNORE ...

And have a UNIQUE (or PRIMARY ) that will catch the "duplicate".

INSERTing one row at a time is about 10 times as slow as inserting 100 rows at a time. So, if you are generating them by code, do

INSERT IGNORE INTO t
    (col1, col2, ...)
    VALUES
    (1,2,...),
    (22,55,...),
    ... ;

Or

LOAD DATA LOCAL INFILE '...' IGNORE ...

if reading from a file.

Answer 2

First thing - do not open a connection for every insert, unless you are using a connection pool.

Second thing - use PreparedStatement . Not only will this save you from SQL injection, it will also make it faster because it will avoid repetitive parsing.

Third thing - use PreparedStatement.addBatch() and commit a batch every 5000 rows (or something like that). This implies you use the same Connection and PreparedStatement for all inserts.

Fourth thing - if you are only filling the database with test data and you know that your test data is unique, create index AFTER you insert all the records. It will be significantly faster.

Fifth thing - if you are using InnoDB, make sure you have enough buffer space to keep entire index in memory, and put the database on SSD (~30x faster than HDD).

If you can do it outside Java, you can use database's proprietary features for bulk loading, restoring from backups or snapshots. Check what features your database provides.