协助优化以下 MySql 程序

Question

MySql - innodb_version 5.7.33

I working on a stored procedure which will be called periodically (lets say once a month) to populate a table with list of string in one column and with static values in other column. The table also has

• ID column (AUTO_INCREMENT) and
• timestamp column (CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP)

The string is a concatenation of fixed characters plus integers (lets say 10). This integer has to be non repetitive random within the range.

CREATE DEFINER=`db`@`%` PROCEDURE `InsertRandom`(IN NumRows INT, IN MinVal INT, IN MaxVal INT)
BEGIN
    DECLARE i INT;
    DECLARE UniqueId INT(15);
    SET i = 1;
    START TRANSACTION;
    WHILE i <= NumRows DO
        SET UniqueId = concat('ABC', MinVal + CEIL(RAND() * (MaxVal - MinVal)));
        IF  NOT EXISTS (SELECT UNIQUE_ID FROM MY_TABLE WHERE UNIQUE_ID = UniqueId) THEN
            INSERT INTO MY_TABLE (`UNIQUE_ID`, `STATE`, `RANGE_ID`) VALUES (UniqueId, 'new', '100');
        END IF;
        SET i = i + 1;
    END WHILE;
    COMMIT;
END

The range (minVal & maxVal) will be 1 million for every procedure call.

For example,

CALL InsertRandom(1000000, 10000000,11000000);

The table will be purged once in 5 months retaining 1 month of data, so we can assume there will be at about 5 million records at a point when this procedure will executed and the usage of select inside loop is not optimal so please suggest an alternative approach.

(From Comment:)

The goal is to have a table with unique IDs within a given range. These shouldn't be in sequence. The range is in minimum of a million at time and maximum of 10 mill. Chunks of which will be loaded into server's memory for further processing. I'm interested in options to populate this table efficiently.

Answer 1

Your stored procedure will found a bottle neck trying to fill some holes. What about to pre-generate the numbers starting MinVal and insert those in a random order, take a look:

CREATE TABLE numbers(
    `UNIQUE_ID` INT,
    `STATE` VARCHAR(50),
    `RANGE_ID` VARCHAR(50)
);


DELIMITER $$
DROP PROCEDURE IF EXISTS InsertRandom $$
CREATE PROCEDURE InsertRandom (IN NumRows INT, IN MinVal INT)
BEGIN
    DECLARE i INT;
    DECLARE UniqueId INT;
    SET i = 0;
    
    CREATE TEMPORARY TABLE IF NOT EXISTS numbers_tmp(
        UNIQUE_ID int, 
        RAND_VAL double, 
        INDEX(RAND_VAL)
    );
    
    START TRANSACTION;
    
    WHILE i < NumRows DO
        INSERT INTO numbers_tmp(UNIQUE_ID,RAND_VAL) 
        VALUES (i + MinVal ,rand());
        SET i = i + 1;
    END WHILE;
    
    INSERT INTO numbers (`UNIQUE_ID`, `STATE`, `RANGE_ID`) 
    SELECT `UNIQUE_ID`, 'new', '100'
    FROM numbers_tmp
    ORDER BY RAND_VAL; /* <-- THIS  */
        
    COMMIT;
    
    DROP TEMPORARY TABLE numbers_tmp;
END $$
DELIMITER ;


CALL InsertRandom(100, 500);

mysql> select  * from numbers;
+-----------+-------+----------+
| UNIQUE_ID | STATE | RANGE_ID |
+-----------+-------+----------+
|       575 | new   | 100      |
|       523 | new   | 100      |
|       560 | new   | 100      |
|       537 | new   | 100      |
|       526 | new   | 100      |
|       549 | new   | 100      |
|       598 | new   | 100      |
|       552 | new   | 100      |
|       555 | new   | 100      |
|       581 | new   | 100      |
...     

At this point. maybe is a good idea just insert the values in your main table and order by some rand inserted number. I don't know.

Answer 2

1. Insert a bunch of random numbers into a table.
2. Remove from that table any numbers that are currently in use.
3. When you need a number, take the 'next' number from that table, then delete it.
4. Check each night to see if the table is 'nearly' empty. If so, rerun the above steps.

Slightly different:

Setup:

CREATE TABLE Numbers (
    id INT UNSIGNED NOT NULL AUTO_INCREMENT,
    number INT UNSIGNED NOT NULL
    PRIMARY KEY(id),
    UNIQUE(number)    -- avoids dups, see IGNORE below
) ENGINE = InnoDB;

Do this nightly, not monthly:

INSERT IGNORE INTO Numbers (number)
    SELECT FLOOR(9000000 * RAND() + 1000000)  -- for range of 1M to 10M
              AS number
        FROM MainTable   -- any table with lots of rows
        HAVING NOT EXISTS( SELECT 1 FROM MainTable
                WHERE Numbers.number = MainTable.number )
        LIMIT 12345;  -- enough to last 3 days, not a month

To get a new, unique, number (pseudo code):

BEGIN;
$number = SELECT number FROM Numbers LIMIT 1 FOR UPDATE;
DELETE FROM Numbers WHERE number = $number;
COMMIT;

Then proceed to use $number in MainTable.