如何在插入之前使用CSV（平面文件）更新MySQL数据库并验证数据

Question

I have about 40K records that I need to update into my php/mysql application on a daily basis. What is the recommended best approach?

I did some research on load data in file to load the flat-file in a temp MySQL table and run validation on the whole column instead of doing one by one row at a time. Later use the primary key and insert/update that in the database.

However for the data which has MultiSelect options or multiple valid values? How do i validate those before updating them in the database.

Options are delimited using piped.

So for example a multi select data is Color type Valid Values are RED GREEN BLUE BROWN BLACK WHITE

Raw value example case scenarios are

Case 1 Raw Data that needs to be validated RED|GREEN|YELLOW

How can is validate this data in the temp table so RED & GREEN pass thru validation and YELLOW gets stripped out and goes in error log?

How can is do the above validation in Bulk on a csv/tabdelimited flat-file with 40K plus records ?

Answer 1

I like your approach of evaluating columns instead of row-by-row, this seems to me that it would give you a performance boost.

To add to your idea, why not create a function to validate a specific column. For the columns with multiple correct values, you just put your check statement into a switch statement or something inside of that function.

Example:

//Example function validates a field that contains an integer value
function validateField1($x)
{

   //if necessary you could parse apart the value before "switching" it.

   if(isset($x))
   {
      switch($x)
      {
         case 1: //integer 1 is a valid value
            return true;
         case 3: //integer 3 is a valid value
            return true;
         default://for all other values return false
            return false;
      }
   }
}

This way you can evaluate the fields on a case by case basis. In addition to that, if your validate function returns a false value you could programmatically edit the .csv file before uploading it to the database.

I have no idea if this will work for you, but do let us know how it turns out!

Regards,

H