PHP mixed UTF-8 encoding reading CSV

Question

I have a problem where I thought it was easy to Google but it seems to be not that easy. Ok, here is my problem:

I have to read a CSV file which has a bad and different encoding. I'm not able to correct the CSV file before hand, so I have to handle it in my application. So the CSV file could have the following-encodings:

'Ü5' and 'Möbelmarkt' in the same file.

If I decode ( utf8_decode ) the right one is correct and the left (which was correct) is wrong. When I try to find out the encoding ( mb_detect_encoding ) i always get the answer that this is UTF-8.

I still tried the following solutions:

public function convert( $str ) {
    return iconv( "Windows-1252", "UTF-8", $str );
}

and

private function getUmlauteArray() { 
    return array( 'ü'=>'ü', 'ä'=>'ä', 'ö'=>'ö', 'Ö'=>'Ö', 'ß'=>'ß', 'à '=>'à', 'á'=>'á', 'â'=>'â', 'ã'=>'ã', 'ù'=>'ù', 'ú'=>'ú', 'û'=>'û', 'Ù'=>'Ù', 'Ú'=>'Ú', 'Û'=>'Û', 'Ãœ'=>'Ü', 'ò'=>'ò', 'ó'=>'ó', 'ô'=>'ô', 'è'=>'è', 'é'=>'é', 'ê'=>'ê', 'ë'=>'ë', 'À'=>'À', 'Ã'=>'Á', 'Â'=>'Â', 'Ã'=>'Ã', 'Ä'=>'Ä', 'Ã…'=>'Å', 'Ç'=>'Ç', 'È'=>'È', 'É'=>'É', 'Ê'=>'Ê', 'Ë'=>'Ë', 'ÃŒ'=>'Ì', 'Ã'=>'Í', 'ÃŽ'=>'Î', 'Ã'=>'Ï', 'Ñ'=>'Ñ', 'Ã’'=>'Ò', 'Ó'=>'Ó', 'Ô'=>'Ô', 'Õ'=>'Õ', 'Ø'=>'Ø', 'Ã¥'=>'å', 'æ'=>'æ', 'ç'=>'ç', 'ì'=>'ì', 'í'=>'í', 'î'=>'î', 'ï'=>'ï', 'ð'=>'ð', 'ñ'=>'ñ', 'õ'=>'õ', 'ø'=>'ø', 'ý'=>'ý', 'ÿ'=>'ÿ', '€'=>'€' );
}

public function fixeUmlaute($string) {                  
    $umlaute = $this->getUmlauteArray();
    foreach ($umlaute as $key => $value){
        $value = str_replace($key, $value, $string);
    } 
    return $string;
}

and

function valid_utf8( $string ){
    return !((bool)preg_match('~[\xF5\xF6\xF7\xF8\xF9\xFA\xFB\xFC\xFD\xFE\xFF\xC0\xC1]~ms',$string));
}

That are all solutions I found with a Google search to change the encoding...(perhaps this "collection" helps anybody else...) So, how can I really detect the wrong characters or where is it my mistake?

Can anybody give me a hint?

Greetz

V

Answer 1

There is a nice PHP class that can help you with that: https://github.com/neitanod/forceutf8 It will convert any charset to UTF8, and handle the detection for you. Hope it helps.