Safely auto increment MySQL field based on MAX() subquery upon insert

Question

I have a table which contains a standard auto-incrementing ID, a type identifier, a number, and some other irrelevant fields. When I insert a new object into this table, the number should auto-increment based on the type identifier.

Here is an example of how the output should look:

id      type_id     number
1       1           1
2       1           2
3       2           1
4       1           3   
5       3           1
6       3           2
7       1           4
8       2           2

As you can see, every time I insert a new object, the number increments according to the type_id (ie if I insert an object with type_id of 1 and there are 5 objects matching this type_id already, the number on the new object should be 6).

I'm trying to find a performant way of doing this with huge concurrency. For example, there might be 300 inserts within the same second for the same type_id and they need to be handled sequentially.

Methods I've tried already:

PHP

This was a bad idea but I've added it for completeness. A request was made to get the MAX() number for the item type and then add the number + 1 as part of an insert. This is quick but doesn't work concurrently as there could be 200 inserts between the request for MAX() and that particular insert leading to multiple objects with the same number and type_id.

Locking

Manually locking and unlocking the table before and after each insert in order to maintain the increment. This caused performance issues due to the number of concurrent inserts and because the table is constantly read from throughout the app.

Transaction with Subquery

This is how I'm currently doing it but it still causes massive performance issues:

START TRANSACTION;
INSERT INTO objects (type_id,number) VALUES ($type_id, (SELECT COALESCE(MAX(number),0)+1 FROM objects WHERE type_id = $type_id FOR UPDATE));
COMMIT;

Another negative thing about this approach is that I need to do a follow up query in order to get the number that was added (ie searching for an object with the $type_id ordered by number desc so I can see the number that was created - this is done based on a $user_id so it works but adds an extra query which I'd like to avoid)

Triggers

I looked into using a trigger in order to dynamically add the number upon insert but this wasn't performant as I need to perform a query on the table I'm inserting into (which isn't allowed so has to be within a subquery causing performance issues).

Grouped Auto-Increment

I've had a look at grouped auto-increment (so that the number would auto-increment based on type_id) but then I lose my auto-increment ID.

---

Does anybody have any ideas on how I can make this performant at the level of concurrent inserts that I need? My table is currently InnoDB on MySQL 5.5

Appreciate any help!

Update: Just in case it is relevant, the objects table has several million objects in it. Some of the type_id can have around 500,000 objects assigned to them.

Answer 1

Use transaction and select ... for update. This will solve concurrency conflicts.

Answer 2

In Transaction with Subquery

Try to make index on column type_id

I think by making index on column type_id it will speed up your subquery.

Answer 3

 DROP TABLE IF EXISTS my_table;

 CREATE TABLE my_table 
 (id      INT NOT NULL AUTO_INCREMENT PRIMARY KEY
 ,type_id     INT NOT NULL
 );

 INSERT INTO my_table VALUES
 (1,1),(2,1),(3,2),(4,1),(5,3),(6,3),(7,1),(8,2);

 SELECT x.*
      , COUNT(*) rank 
   FROM my_table x 
   JOIN my_table y 
     ON y.type_id = x.type_id 
    AND y.id <= x.id 
  GROUP 
     BY id 
  ORDER 
     BY type_id
      , rank;

 +----+---------+------+
 | id | type_id | rank |
 +----+---------+------+
 |  1 |       1 |    1 |
 |  2 |       1 |    2 |
 |  4 |       1 |    3 |
 |  7 |       1 |    4 |
 |  3 |       2 |    1 |
 |  8 |       2 |    2 |
 |  5 |       3 |    1 |
 |  6 |       3 |    2 |
 +----+---------+------+

or, if performance is an issue, just do the same thing with a couple of @variables.

Answer 4

Perhaps an idea to create a (temporary) table for all rows with a common "type_id". In that table you can use auto-incrementing for your num colomn. Then your num shoud be fully trustable. Then you can select your data and update your first table.