Issue with file_get_contents encoding

Question

I'm getting file_get_contents(uri) and getting back a Json that I'm unable to encode.

I tried several encodings and str_replace but I don't quite understand what the issue is.

This is the start of my json with file_get_contents :

string(67702) " {"localidades"

I know it's finding unknown characters and that's what the ? are for, but I don't understand how to solve it.

I've tried this but to no avail

    if(substr($s, 0, 2) == chr(0xFF).chr(0xFE)){
      return substr($s,3);
    }
    else{
      return $s;
    }
   }

This is xxd | head from terminal

00000000: fffe 7b00 2200 6c00 6f00 6300 6100 6c00  ..{.".l.o.c.a.l.
00000010: 6900 6400 6100 6400 6500 7300 2200 3a00  i.d.a.d.e.s.".:.
00000020: 2000 5b00 7b00 2200 6900 6400 4c00 6f00   .[.{.".i.d.L.o.
00000030: 6300 6100 6c00 6900 6400 6100 6400 2200  c.a.l.i.d.a.d.".
00000040: 3a00 2000 3300 2c00 2200 6c00 6f00 6300  :. .3.,.".l.o.c.
00000050: 6100 6c00 6900 6400 6100 6400 2200 3a00  a.l.i.d.a.d.".:.
00000060: 2000 2200 4200 7500 6500 6e00 6f00 7300   .".B.u.e.n.o.s.
00000070: 2000 4100 6900 7200 6500 7300 2200 2c00   .A.i.r.e.s.".,.
00000080: 2200 6900 6400 5000 7200 6f00 7600 6900  ".i.d.P.r.o.v.i.
00000090: 6e00 6300 6900 6100 2200 3a00 2000 2200  n.c.i.a.".:. .".

Answer 1

What you have there is UTF-16LE in which each codepoint is encoded as at least two bytes, even "basic ASCII". The first two bytes of the document are the Byte Order Mark [BOM] that declares in what byte-order [endian] those codepoints are encoded

$input = "\xff\xfe{\x00}\x00"; // UTF-16-LE with BOM

function convert_utf16($input, $charset=NULL) {
    // if your data has no BOM you must explicitly define the charset.
    if( is_null($charset) ) {
        $bom = substr($input, 0, 2);
        switch($bom) {
            case "\xff\xfe":
                $charset = "UTF-16LE";
                break;
            case "\xfe\xff":
                $charset = "UTF-16BE";
                break;
            default:
                throw new \Exception("No encoding specified, and no BOM detected");
                break;
        }
        $input = substr($input, 2);
    }
    return mb_convert_encoding($input, "UTF-8", $charset);
}

$output = convert_utf16($input);

var_dump(
    $output,
    bin2hex($output),
    json_decode($output, true)
);

Output:

string(2) "{}"
string(4) "7b7d"
array(0) {
}

It's also worth noting that using anything other than UTF-8 to encode JSON makes it invalid JSON, and you should tell whoever is giving you this data to fix their app.

Answer 2

What you are getting is UTF-16 LE . The fffe at the beginning is called a BOM . You can use iconv :

$data = iconv( 'UTF-16', 'UTF-8', $data);

And now you have a UTF-8 with BOM . Which i think will work with json_decode , because PHP seems to handle it. Still, if you want to remove the BOM , which you should (see @Sammitch comment), you can use this one as well:

$data = preg_replace("/^pack('H*','EFBBBF')/", '', $data);

I recreated a part of your file and i get this:

$data = file_get_contents('/var/www/html/utf16le.json');
$data = preg_replace("/^pack('H*','EFBBBF')/", '', iconv( 'UTF-16', 'UTF-8', $data));
print_r(json_decode($data));

Output:

stdClass Object
(
    [localidades] => Array
        (
            [0] => stdClass Object
                (
                    [idLocalidad] => 3
                    [localidad] => Buenos Aires
                )

        )

)

And from xxd :

Answer 3

The file you try to process is encoded in UTF-16, which isn't natively supported by PHP. So, in order to process it, you'll have to remove BOM header first (first two bytes) and then convert encoding to UTF-8 using iconv or mbstring.