Javascript encodeURI returns unexpected value

Question

I have a problem URL-encoding a text with javascript. I am in Germany, where we have these "Umlaute" (ÄÖÜ), and these letters make some problems.

An online encoder/decoder returned the following results for the word "Äpfel" (apples).

Äpfel >>> url-encode >>> %C3%84pfel

%C3%84pfel >>> url-decode >>> Äpfel

For testing, I created the following php.file (poc.php) with no php-content, just the javascript:

<script type="text/javascript">
    var t = "Äpfel";
    t = encodeURI(t);
    alert(t);
    t = decodeURI(t);
    alert(t);
</script>

The first alert returns "%EF%BF%BDpfel", which differs from the result of the online encoder.

The second alert returns " pfel" (yes, the diamond with the "?").

It seems that javascript cannot decode the text it just encoded.

I guess the cause of this behaviour is somewhere in the PHP settings. When I just rename the file from "poc.php" to "poc.html" the encoding is correct and the alerts return the same results as the online encoder/decoder.

When I check the current encoding, javascript and php return "utf-8".

In my "real" project I have a ".js" file included in my php-file (with the same problem).

<script type="text/javascript" src="scripts/functions.js"></script>

Has anybody an idea what causes this behaviour?

Answer 1

The weird byte stream %EF%BF%BD you're receiving is utf-8 version of the Unicode replacement character, that is, literally the symbol.

The Javascript portion can url-decode the text it just url-encoded, it was just asked to encode the symbol for a missing symbol.

So: some part of your system is not using utf-8, but some other character set instead, and there's an unnecessary conversion done. My guess is that the file is encoded in latin-1, aka. ISO 8859-1, and PHP tries to read it as if it was UTF-8, converting the unrecognized character 0xc4 ('Ä' in latin-1) to the replacament character symbol.