UTF-8 escaping with HTMLentities returns blank

Question

I am importing a CSV fil**e into my database using **MySQL'S LOAD DATA INFILE command. The file is not necessarily UTF-8 encoded , I don't have any control over that so i must resort to pre/post processing. Both my database and HTML web pages enforce UTF-8 encoding . Since I do Load Data infile, I have to post process this. So I extract the information out of the database and apply my post processing filters using htmlentities.

   foreach($records as $r)
       $updates[] = htmlentities($r["column"], ENT_COMPAT, 'UTF-8');

Then I update the DB table again.

The columns go into the table just as they should before the post processing but after that, the columns go blank which means htmlentities returned a blank and that is a valid return value for HTML.

Specifically a candidate value is:

"PJ Weatherproof 32 ®"

Any idea why ?

Answer 1

The "UTF-8" parameter in your call promises htmlentities() that the incoming data will be UTF-8. When the data isn't, which will cause the function to return a blank value.

You'll need to try and sniff the encoding, which is an unreliable process and will work well only when you have a very limited set of possible encodings. See eg this answer.

Either way, you can get rid of the htmlentities() call - it will do nothing to help the situation, just uselessly add HTML entities where they aren't needed.