Java-MySQL: How to create a Stored Procedure

Question

I Work on a Java 1.7 project with Mysql,

I have a method that insert a lot of data in a table with PreparedStatement, but this cause a Out Of Memory Error in the GlassFish Server.

Connection c = null;
    String query = "INSERT INTO users.infos(name,phone,email,type,title) "
            + "VALUES (?, ?, ?, ?, ?, ";
    PreparedStatement statement = null;
    try {
        c = users.getConnection();
        statement = c.prepareStatement(query);
    } catch (SQLException e1) {
        // TODO Auto-generated catch block
        e1.printStackTrace();
    }

try{
        int i =0;
        for(Member member: members){
            i++;
            statement.setString(1, member.getName());
            statement.setString(2, member.getPhone());
            statement.setString(3, member.getEmail());
            statement.setInt(4, member.getType());
            statement.setString(5, member.getTitle());
            statement.addBatch();
            if (i % 100000 == 0){
                statement.executeBatch();
            }
        }
        statement.executeBatch();
    }catch (Exception ex){
        ex.printStackTrace();
    } finally {
        if(c != null)
        {
            try {
                c.close();
            } catch (SQLException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }
        }
        if(statement != null){
            try {
                statement.close();
            } catch (SQLException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }
        }
        c = null;
        statement = null;
    }

I think that I need to create a Stored Procedure to avoid this memory issue, but I don't know where to start and if I should create a procedure or a function, and if I will be able to get some kind of response in a return or something?

What do you think about it?

Answer 1

Replacing your insert statement in your prepared statement with a call to a stored procedure in your prepared statement will not affect the memory consumption in any meaningful way.

You are running out of memory because you are using a very large batch size. You should test the performance of different batch sizes - you will find that the performance improvement for larger batches diminishes quickly with batch sizes greater than dozens to hundreds.

You may be able to achieve greater input rates by using load data infile . You would take your many rows of data, create a text file and loading the file. For example, see this question .

You may also consider parallelizing. For example, open multiple connections and insert rows to each connection with separate threads. Likewise, you could try doing parallel loads using load data infile .

You will have to try the various techniques, batch sizes, number of threads (probably not more than one per core), etc. on your hardware setup to see what gives the best performance.

You may also want to look at tuning some of the MySQL parameters, drop (and later recreate) indexes, etc.