client or server is removing umlauts

Question

I am using btoa to encode my string to base64. However, it does not seem to work correctly, any umlauts are removed for whatever reason.

I tried to demonstrate it, but on this page it works:

 var content = "This text contains umlauts (ÖÜÄ) they will be removed on my webpage, but not in this demo for whatever reason"; alert(content); content = btoa(content); alert(content); content = atob(content); alert(content); 

But it does not work on my webpage:

This is the value of the parameter content :

PHA+VGhpcyB0ZXh0IGNvbnRhaW5zIHVtbGF1dHMgKMTc1ikuIExldHMgc2VlIGlmIHRoZXkgYXJlIGdldHRpbmcgcmVtb3ZlZC48L3A+

If you decode this base64 encoded string here then you will get this result:

This text contains umlauts (). Lets see if they are getting removed.

As you can see the umlauts where removed for whatever reason.

This is a snippet from the code where I encode the string, it is used to post a news entry to a blog:

---

EDIT : The code below suddenly works without any change. See other edit below for more infos.

---

var content = $(".ql-editor").html();

$.ajax({
    url: "ajax.php",
    method: "POST",
    data: {
        action: "postNews",
        autor: autor,
        content: btoa(content),
        date: date
    },
    success: function(response) {

        if (response.indexOf("error") === -1) {
            $("#statusBoxMessage").html("Ihr Eintrag wurde erfolgreich veröffentlicht.");
            $("#statusBoxImage").attr("src", "gfx/page/checkmark.jpg");
            showStatusBox("success");
        } else {

            $("#statusBoxMessage").html("Fehler! Der Eintrag konnte nicht erstellt werden! (" + response + ")");
            $("#statusBoxImage").attr("src", "gfx/page/fail.jpg");
            showStatusBox("fail");  
        }
    },
    error: function(response) {
        $("#statusBoxMessage").html("Fehler! " + response);
        showStatusBox("fail");
    }
});

---

EDIT: I also have an almost identical script which I use to edit a news entry from the blog, it does still not work and drops any umlauts:

---

$("#submitEditContent").click(function() {

    var content = $("#editEditor").val();
    var s = 0;

    if (target == "pages/news.html" || target == "news.html") {
        target = currentNewsFilename;
        s = 1;
    }

    $.ajax({
        url: "ajax.php",
        method: "POST",
        data: {
            action: "postEdit",
            content: btoa(content),
            target: target
        },
        success: function(response) {
            if (response == "success") {
                if (s == 0) {
                    window.location.replace("index.php?page=" + target);
                } else {
                    window.location.reload();
                }

            } else {
                console.warn(response);
            }
        }
    });

I hope someone can explain why the first script suddenly stopped dropping the umlauts and why the second script still drops umlauts.

I tested it with the latest chrome browser.

Answer 1

 let decoded = atob('PHA+VGhpcyB0ZXh0IGNvbnRhaW5zIHVtbGF1dHMgKMTc1ikuIExldHMgc2VlIGlmIHRoZXkgYXJlIGdldHRpbmcgcmVtb3ZlZC48L3A+') console.log(decoded) 

I don't think there's anything wrong with the encoding on your webpage.

This works perfectly fine in the browser. And it works on the external site you referenced, but you need to set the encoding to ISO-8859-1 instead of UTF-8. Give it a try.

Answer 2

I solved it thanks to @Matt Morgans's hints. We found out that everything was working fine client side, I had to change server side code.

Before:

$content = $_POST["content"];
$content = stripScript(base64_decode($content));
$content = mb_convert_encoding($content, 'UTF-8', 'OLD-ENCODING');

After:

$content = $_POST["content"];
$content = stripScript(base64_decode($content));
$content = mb_convert_encoding($content, 'UTF-8', 'Windows-1252');

This works too:

$content = mb_convert_encoding($content, 'UTF-8', 'ISO-8859-1');