How to set utf8 character set in magento core resource?

Question

I working on bulk product import from API response.This bulk product import will be handle huge data update using mysql query of core resource connection.

So in this case system will receive some of special character from the Api Response, That special characters should be like below.

[Name] => GÄNGT M8X0.75 6H

We need to save this value should be like GÄNGT M8X0.75 6H .

For the reason of bulk update we are using direct update query to hit the mysql database instead of using native magento adapter.

These above special character not updating with utf8 conversion while doing direct update.But if we use magento product import adapter it will convert and save as value in mysql database.

I have tried to add set character_set_results=utf8 in magento core resource collection, but there is no luck.

Below is my try out :

$resource = Mage::getSingleton('core/resource');
$writeConnection = $resource->getConnection('core_write');
$writeConnection->query("set character_set_results=utf8"); 
$writeConnection->query($mysqlUpdateQuery);
$writeConnection->closeConnection();

Can any one help me, what goes wrong or what i want to add / modify for utf8 value conversion.

Any help much appreciation!

Answer 1

Ä is the Mojibake for utf8 Ä .

Usually Mojibake occurs when

• The bytes you have in the client are correctly encoded in utf8 (good).
• You connected with SET NAMES latin1 (or set_charset('latin1') or ...), probably by default. (It should have been utf8 .)
• xx The column in the table was declared CHARACTER SET latin1 . (Or possibly it was inherited from the table/database.) (It should have been utf8 .)
• The column in the tables may or may not have been CHARACTER SET utf8 , but it should have been that.

Since these seem to disagree with what you said, let's dig further. Please provide

SELECT col, HEX(col) FROM ... WHERE ...

GÄNGT M8X0.75 6H , if correctly stored in utf8 will have hex 47 C384 4E4754204D3858302E3735203648 (I added spaces);
If stored incorrectly (in one way), the hex will be 47 C383 E2809E 4E4754204D3858302E3735203648 .

Do you see either of those? Or a third hex?

With that answer, we can proceed to plan corrective actions.

C383 E2809E was stored

That probably happened thus. And the result was "double-encoding", not "Mojibake".

• The client had C384 , the correct utf8 encoding for Ä .
• The initialization was incorrectly set to latin1 . This needs to be changed. Note that you had $writeConnection->query("set character_set_results=utf8"); , which only handles the output side, not the input side. Read about SET NAMES . Change it to $writeConnection->query("SET NAMES utf8");
• The column was correctly declared CHARSET utf8 .

To repair the data:

UPDATE tbl SET name = CONVERT(BINARY(
                        CONVERT(name USING latin1))
                      USING utf8);

Answer 2

To set utf8_general_ci Mysql Database character set in magento as below

After you have created the database, you need to run this sql query:

ALTER DATABASE DB_NAME DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;

Where DB_NAME is your database name.