How to LOAD DATA with optional fields?

Question

Is it possible to LOAD DATA a csv into mysql without having to add empty values for non existing columns at the end?

All my optional columns are sorted at the end of the schema:

CREATE TABLE `person` (
    id int(20) NOT NULL AUTO_INCREMENT,
    firstname varchar(30) NOT NULL,
    lastname varchar(30) NOT NULL,
    optional1 varchar DEFAULT NULL,
    optional... varchar DEFAULT NULL,
    optional50 varchar DEFAULT NULL,
    PRIMARY KEY (`id`)
) engine=innodb AUTO_INCREMENT=0;

sample.csv:

1;john;doe
2;jabe;doe;;;opt val3;;;;;;opt val9;;;;;;...

Important: I don't want to explicit list all the columns in my LOAD DATA INFILE sql statement (I know that this would work by using a combination of IFNULL and @var ).

But can't I just load into the table, telling mysql to ignore any missing fields at the end of each line?

Answer 1

The documentation of MySQL LOAD DATA syntax provides the following information:

By default, when no column list is provided at the end of the LOAD DATA statement, input lines are expected to contain a field for each table column. If you want to load only some of a table's columns, specify a column list.

[...]

If an input line has too few fields, the table columns for which input fields are missing are set to their default values. For numeric types, the column is set to 0 .

[...]

An empty field value is interpreted different from a missing field: for string types, the column is set to the empty string.

So given your sample data:

1;john;doe
2;jabe;doe;;;opt val3;;;;;;opt val9;;;;;;...

Record with id 1 will have all optional columns set to NULL (ie their default). For id 2, optional string columns will be set to the empty string. .

I cannot tell whether this would be OK for your use case or not. If you do want consistent values in the optional columns, available options would be:

• input pre-processing: use SET to set to NULL columns that contains an empty string

    LOAD DATA INFILE 'file.txt' INTO TABLE t1
    SET 
        optional1 = NULLIF(optional1, ''), 
        optional2 = NULLIF(optional1, ''), 
        ...

• set up a BEFORE INSERT trigger on the table that sets to NULL empty values

• run an update on the table after it was populated

    UPDATE t1 SET optional1 = NULLIF(option1, ''), optional2 = NULLIF(optional1, '')
    WHERE '' IN (optional1, optional2, ...)

Answer 2

I found out it works as expected if adding the IGNORE keyword to the LOAD DATA statement:

LOAD DATA INFILE 'sample.csv' IGNORE INTO TABLE persons

Thereby, I can define all my optional columns as DEFAULT NULL , and if values are missing, they are set to NULL during import.