mysql 定价表中大量连续（重新）导入和刷新数据

Question

I have a large dataset (~2.5 million rows), which needs to be (re-)imported continously into a MySQL table "price_list". All tables are InnoDB. Currently I'm using "LOAD DATA LOCAL INFILE", because those datasets come from csv files:

LOAD DATA LOCAL INFILE 'sample.csv'
INTO TABLE `price_list`
(...)
(...)
IGNORE 1 LINES

Example of a table in in db "price_list":

hotel_id | room_category            | price 1 person | price 2nd person     | <other meta info>

1        | single room (w/o window) | 150€           | 200€                 | ...
2        | single room (w window)   | 170€           | 220€                 | ...
3        | single room (rooftop)    | 240€           | 250€                 | ...
4        | single room (whirlpool)  | 200€           | 280€                 | ...
5        | double room (w/o window) | 200€           | 220€                 | ...
6        | double room (w window)   | 240€           | 260€                 | ...
7        | double room (rooftop)    | 280€           | 300€                 | ...
8        | double room (whirlpool)  | 320€           | 340€                 | ...
(...)

Based on this data I need to update table "offers" (the table sits in another database, user of has "price_list" no access to "offers" and its technically not possible to give the proper access) with refreshed pricings. Pricings change a lot and we need to reimport those data every 15 minutes.

id  |       offer_name           |   price_single_room    |    price_double_room
1   |  WHIRLPOOL OFFER SINGLES   |         200€           |          200€

In the above example the price for the "best" single room (with a whirlpool) was choosen (200€). The 2nd price is not needed in this offer, but is calculated on purpose (and can be deactived if wanted).

My current solution is that I fetch all offers from the "offers" table in PHP, which are marked as active, and loop through them ( oof ). Each offer has 6 columns for different pricings (eg hotel has different rooms available; column 1 is the best price for single room, column 2 is the best price for double room, ...) that needs to be looked up.

Currently we have around 10.000 active offers which means I'm sending the following amount to the database server: 10.000 queries * 6 queries to look up best pricings per offer.

When I execute those queries on the same server (no network latency etc.) the performance is not the worst (the whole job takes around 5 minutes (importing ~2.5m rows, refresh pricings, ...)), but since the data is growing, we want to split up database and webserver. I now realized that the part where I'm refreshing the prices produces a lot of overhead with network and is very slow since each request from webserver to db server takes around 0.025s (25minutes only for refreshing prices).

I thought about the following solution(s):

• move the table "offers" to the same database "price_list" = working, but still very slow since the database server is not on the same machine as the webserver = network latency is the bottleneck.

• write a stored procedure, which is triggered by PHP and the database server does the job.

Does someone have experience with those recurring data loads and maybe a solution for my given problem? The goal is to reduce timings and split web and database server.

Thanks!

Answer 1

You have a table called `real; the following will replace it with a new table containing the fresh data.

CREATE TABLE t_new LIKE real;
LOAD DATA INFILE new ...;
RENAME TABLE real TO t_old,
             t_new TO real;
DROP TABLE t_old;

Notes:

⚈  The LOAD DATA step can be replaced by whatever process you have for importing the data.
⚈  The Loading is the only slow step.
⚈  The RENAME is atomic, so real always exists.
⚈  You may choose to delay the DROP in case the new data might be bad and you want to revert.
⚈  FOREIGN KEYs can be a hassle; it might be good not to have such.

-- http://mysql.rjweb.org/doc.php/deletebig#optimal_reload_of_a_table