Mass MySQL updates

Question

I've written an application which servers can use. This application gathers information, and sends it to a server. This is done every 10 seconds. The amount of data depends on players playing, but lets keep it at about 50 servers each sending 100 pieces of data (5000 total every 10 seconds).

Those pieces of data exist of an SQL query (in PreparedStatement syntax), an Object[] of values for the SQL query, and the serverID.
Now, I want to process all the data. And this is what's not going well for me. I have a MySQL server, which has 5 tables. Three of the tables are constantly updating, and can't keep up with the data flow. The tables are InnoDB tables, mostly since I can row-level lock that way, instead of table-lock. Most of the queries are UPDATE queries anyway, barely any INSERT statements.

I've tried executing all queries right away, which just resulted in horrible performance since the connecting server also had to wait for that to be finished.
I've also tried putting all queries in a big ConcurrentLinkedQueue , and emptying this queue every few seconds. The performance was better, but this was way too slow.
Then I tried a per-table solution, which was slightly better again. Way too slow still.
Currently, it's using a per-server setup (Creates a new thread for every server and executes all queries there). This still is too slow. It can't keep up.

As you see, I've tried numerous things. I've also tried using addBatch() followed by executeBatch() , which is also used in the current setup. Of course, I've also looked on here, Google, etc. Has some useful info, but mostly it's just about adding PreparedStatements and using BatchUpdates.

Any ideas on how to do this?

Answer 1

I used spring-jdbc, and I use com.jolbox.bonecp for connections . I would like to recommend you to use same.

I use jdbcTemplate.batchUpdate(query, multyPreparedValues) where multyPreparedValues is List<Object[]>

but you also can use BatchPreparedStatementSetter — example https://stackoverflow.com/a/8873528/814304