加速本地MySql在大表上启动规范化查询

Question

I'm normalizing and "cleaning" a MySql database wherein the biggest table counts ~ 3 mln records.

What I have to do is to rename some fields (very fast), change their order (quite fast), and doing some trim, string sanitization, extraction of some to other tables and keep the foreign key id...

Is there a way so I can speed up the query on my local machine?

I've MariaDB 10.1.21 (from XAMPP), and running on a MacBook Air 8GB Ram.

I've already put indexes on many fields but it's still slow as a turtle.

Any tip will be appreciated. Thanks!

Edit: as requested more info and some optimization I am performing.

I've basically a big table that contains columns not normalized that would normally been distributed in three tables.

What I have:

companies ( id, name, street, city_name, category_name, subcategory_name )

what I want

companies ( id, name, street, id_city, id_subcategory, ... )
cities( id, name, ... )
categories( id, name )
subcategories( id, name, id_category )

So i clean and exctract the datas as follow.

Trim and clean carriage returns from "dirty" fields:

update companies set mic_cat = TRIM(REPLACE(REPLACE(mic_cat, '\r', ''), '\n', ''));

Delete companies that hasn't a correct category.

delete from companies where mic_cat is null or mic_cat = '' or mac_cat is null or mac_cat = '';

Extract the data from the fields and place in new tables:

insert into categories (name) select distinct mac_cat from companies;
insert into subcategories (name, id_category) select distinct mic_cat,categories.id from companies JOIN categories ON mac_cat = categories.name;

Add the id_reference:

ALTER TABLE companies ADD COLUMN id_subcategory int;

Get the keys...

UPDATE companies left join subcategories on companies.mic_cat = subcategories.name set id_subcategory = subcategories.id;

The last one was very slow, so, I dropped all the indexes and then create just two index on companies.mic_cat and subcategories.name and it has been fastened quite a bit.

Answer 1

• Do all updates in a single UPDATE statement.
• If you need to modify columns that are in index(es), DROP those indexes first and ADD back later. (This may help.)
• Do all ALTERs in a single ALTER statement. (This is not always the best advice.)
• Think about doing the updates in chunks of rows.

Some issues that the above tries to address:

• UPDATE without a WHERE clause (and sometimes with a WHERE ) will scan through the entire table, being rather costly.
• When an indexed column is modified, the row in the index needs to be removed from one place in the index and added in another. Think of it as a DELETE plus an INSERT -- rather costly.
• ALTER may or may not be able to do the work "in place". If multiple of your alters cannot be done that way, then it is best to do a single copy (ie, a single ALTER ) to do all the changes simultaneously. It effectively creates a new empty table, alters it, copies all the data into it, recreates all the indexes, then renames it back into place.

More on indexes...

• Don't index flags; such indexes will be shunned.
• Look at your WHERE clauses to see what indexes would be useful.
• Learn about 'composite' indexes: INDEX(a,b) may be much better than INDEX(a), INDEX(b) for some queries.
• Don't blindly index every column -- a big waste.

3M rows is possibly a lot. In many situations, it is better to UPDATE (or DELETE ) in "chunks". See my blog .