使用php将大量excel导入mysql

Question

I am having excel sheet which contains data in something like this ex. if sheet has "city" and "person name" columns.So every time we need to check whether the city exists in city table(mysql) or not if not then new row will be inserted for city in city table similarly will check for person name also if don't exists in person table(mysql) then we need to save person name in person table . So what can be the best way to import heavy excel file of 2gb atleast into mysql using php in the above explained scenario so that less queries will be executed with less load on server and execution will be done in fraction of seconds. Kindly suggest me the optimize solution for above scenario.

Answer 1

For high speed ingestion that needs to "normalize" the data as it goes, do it in two steps (after loading the raw data into a temp table):

First, add the new values (of host_name in this example):

# This should not be in the main transaction, and it shoud be with autocommit = ON
# In fact, it could lead to strange errors if this were part of the main transaction and it ROLLBACKed.
INSERT IGNORE INTO HostNorm (host_name)
    SELECT DISTINCT s.host_name
        FROM Staging AS s
        LEFT JOIN HostNorm AS n  ON n.host_name = s.host_name
        WHERE n.host_id IS NULL;

Then grab the host_id for putting into the Fact table:

# Also not in the main transaction, and it should be with autocommit = ON
# This multi-table UPDATE sets the ids in Staging:
UPDATE   HostNorm AS n
    JOIN Staging AS s  ON s.host_name = n host_name
    SET s.host_id = n.host_id

More discussion in my blog