mb_strlen() & strlen() don't return correct values from an Ajax call to PHP

Question

How can I add a check in the PHP for the length of the $username passed. The site is UTF-8 but I believe Javascript is using a different encoding. You can see in the comments where I tried different things in the PHP and they don't work.

What I tried and didn't work:

• Changing Ajax (javascript) to pass variables by UTF-8 and not javascript encoding
• strlen, mb_strlen in the PHP - both return incorrect values

MORE INFO

My Ajax sends a username to my PHP, which checks the SQL DB and returns available or not. I decided to try and do some extra checking in the PHP before checking the DB (like mb_strlen($username ). mb_internal_encoding("UTF-8"); is also set.

I was going to try and send the Ajax request in UTF-8 but didnt see a way to do that.

is UPPER being used correctly in the MySQL? - for UTF-8 stuff?

PHP BELOW * ** * ** * ** * *

// Only checks for the username being valid or not and returns 'taken' or 'available'
require_once('../defines/mainDefines.php'); // Connection variables
require_once('commonMethods.php');
require_once('sessionInit.php');    // start session, check for HTTP redid to HHHTPs

sleep(2);   // Looks cool watching the spinner

$username = $_POST['username'];

//if (mb_strlen($username) < MIN_USERNAME_SIZE) echo 'invalid_too_short';

//if (mb_strlen($username, 'UTF-8') < 10) { echo ('invalid_too_short'); exit; }
//die ('!1!' .  $username . '!2!' . mb_strlen($username) . '!3!' . strlen($username) . '!4!');

$dbc = mysqli_connect(DB_HOST, DB_READER, DB_READER_PASSWORD, DB_NAME) or     die(DB_CONNECT_ERROR . DB_HOST .  '--QueryDB--checkName.php');
$stmt = mysqli_stmt_init($dbc);

$query = "SELECT username FROM pcsuser WHERE UPPER(username) = UPPER(?)";
if (!mysqli_stmt_prepare($stmt, $query)) {
    die('SEL:mysqli_prepare failed somehow:' . $query . '--QueryDB--checkName.php');
}

if (!mysqli_stmt_bind_param($stmt, 's', $username)) {
    die('mysqli_stmt_bind_param failed somehow --checkName.php');
}

if (!mysqli_stmt_execute($stmt)) {
    die('mysqli_stmt_execute failed somehow' . '--checkName.php');
}

mysqli_stmt_store_result($stmt);
$num_rows = mysqli_stmt_num_rows($stmt);
mysqli_stmt_bind_result($stmt, $row);           
echo ($num_rows >= 1) ? 'taken' : 'available';

mysqli_stmt_close($stmt);
mysqli_close($dbc);

AJAX CODE BELOW

function CheckUsername(sNameToCheck) {
document.getElementById("field_username").className = "validated";
registerRequest = CreateRequest();
if (registerRequest === null)
    alert("Unable to create AJAX request");
else {
  var url= "https://www.perrycs.com/php/checkName.php";
  var requestData = "username=" + escape(sNameToCheck); // data to send
  registerRequest.onreadystatechange = ShowUsernameStatus;
  registerRequest.open("POST", url, true);
  registerRequest.setRequestHeader("Content-Type","application/x-www-form-urlencoded");
  registerRequest.send(requestData);
}
}


function ShowUsernameStatus() {
var img_sad = "graphics/signup/smiley-sad006.gif";
var img_smile = "graphics/signup/smiley-happy088.gif";
var img_checking = "graphics/signup/bluespinner.gif";

if (request.readyState === 4) {
    if (request.status === 200) {
        var txtUsername = document.getElementById('txt_username');
        var fieldUsername = document.getElementById('field_username');
        var imgUsername = document.getElementById('img_username');
        var error = true;
        var response = request.responseText;

        switch (response) {
            case "available":
                txtUsername.innerHTML = "NAME AVAILABLE!";
                error = false;                  
                break;
            case "taken":
                txtUsername.innerHTML = "NAME TAKEN!";
                break;
            case "invalid_too_short": 
                txtUsername.innerHTML = "TOO SHORT!";
                break;
            default:
                txtUsername.innerHTML = "AJAX ERROR!";
                break;
        } // matches switch

        if (error) {
            imgUsername.src = img_sad;
            fieldUsername.className = 'error';
        } else {
            imgUsername.src = img_smile;
            fieldUsername.className = 'validated';
        }
    } // matches ===200
} // matches ===4
}

TESTING RESULTS

This is what I get back when I DIE in the PHP and echo out as in the following (before and after making the Ajax change below [adding in UTF-8 to the request]...

PHP SNIPPIT

die ('!1!' .  $username . '!2!' . mb_strlen($username) . '!3!' . strlen($username) . '!4!');

TEST DATA

Username: David Perry

!1!David Perry!2!11!3!11!4!

Username: ܦ"~÷Û♦

!1!ܦ\\"~ %u2666!2!9!3!13!4!

The first one works. The second one should work but it looks like the encoding is weird (understandable).

7 visible characters for the 2nd one. mb_strlen shows 9, strlen shows 13.

After reading Joeri Sebrechts solution and link they gave me I looked up Ajax request parameters and someone had the following...

AJAX SNIPPIT (changed from original code)

registerRequest.setRequestHeader("Content-Type","application/x-www-form-urlencoded; charset=UTF-8");

(I added in the charset=UTF-8 from an example I saw on a article).

UPDATE: Nov 27, 9:11pm EST

Ok, after much reading I believe I am encoding my JS wrong. I was using escape... as follows...

var requestData = "username=" + escape(sNameToCheck);

After looking at this website...

http://www.the-art-of-web.com/javascript/escape/

it helped me understand more of what's going on with each function and how they encode and decode. I should be able to do this...

var requestData = "username=" + encodeURIComponent(sNameToCheck);

in JS and in PHP I should be able to do this...

$username = rawurldecode($_POST['username']);

Doing that still gives me 8 characters for my weird example above instead of 7. It's close, but am I doing something wrong? If I cursor through the text on the screen it's 7 characters. Any ideas to help me understand this better?

FIXED/SOLVED!!!

Ok, thank you for your tips that lead me in the right direction to make this work. My changes were as follows.

In the AJAX -- i used to have escape(sNameToCheck); --

var requestData = "username=" + encodeURIComponent(sNameToCheck);

In the PHP *-- I used to have $username = $_POST['username']; --*

$username = rawurldecode($_POST['username']);
if (get_magic_quotes_gpc()) $username = stripslashes($username);

I really hate magic_quotes... it's caused me about 50+ hours of frustration over form data in total because I forgot about it. As long as it works. I'm happy!

So, now the mb_strlen works and I can easily add this back in...

if (mb_strlen($username) < MIN_USERNAME_SIZE) { echo 'invalid_too_short'; exit; }

Works great!

Answer 1

PHP is a byte processor, it is not charset-aware. That has a number of tricky consequences.

Strlen() returns the length in bytes, not the length in characters. This is because php's "string" type is actually an array of bytes. Utf8 uses more than one byte per character for the 'special characters'. Therefore strlen() will only give you the right answer for a narrow subset of text (= plain english text).

Mb_strlen() treats the string as actual characters, but assumes it's in the encoding specified via mbstring.internal_encoding, because the string itself is just an array of bytes and does not have metadata specifying its character set. If you are working with utf8 data and set internal_encoding to utf8 it will give you the right answer. If your data is not utf8 it will give you the wrong answer.

Mysql will receive a stream of bytes from php, and will parse it based on the database session's character set, which you set via the SET NAMES directive. Everytime you connect to the database you must inform it what encoding your php strings are in.

The browser receives a stream of bytes from php, and will parse it based on the content-type charset http header, which you control via php.ini default_charset. The ajax call will submit in the same encoding as the page it runs from.

Summarized, you can find advice on the following page on how to ensure all your data is treated as utf8. Follow it and your problem should resolve itself. http://malevolent.com/weblog/archive/2007/03/12/unicode-utf8-php-mysql/

Answer 2

From a quick glance, you can clean this up:

if (request.status == 200) {
    if (request.responseText == "available") {
        document.getElementById("txt_username").innerHTML = "NAME AVAILABLE!";
        document.images['img_username'].src=img_smile;
        document.getElementById("continue").disabled = false;
        document.getElementById("field_username").className = 'validated';
    } else if (request.responseText == "taken") {
        document.getElementById("txt_username").innerHTML = "NAME TAKEN!";
        document.images['img_username'].src=img_sad;
        document.getElementById("field_username").className = 'error';
    } else if (request.responseText == "invalid_too_short") {
        document.getElementById("txt_username").innerHTML = "TOO SHORT!";
        document.images['img_username'].src=img_sad;
        document.getElementById("field_username").className = 'error';
    } else {
        document.getElementById("txt_username").innerHTML = "AJAX ERROR!";
                document.images['img_username'].src=img_sad;
        document.getElementById("field_username").className = 'error';
    }
  }

to:

// I prefer triple equals
// Read more at http://javascript.crockford.com/style2.html
if (request.status === 200) {
        // use variables!
        var txtUsername = document.getElementById('txt_username');
        var fieldUsername = document.getElementById('field_username');
        var imgUsername = document.getElementById('img_username');

        var response = request.responseText;

        var error = true;

        // you can do a switch statement here too, if you prefer
        if (response === "available") {
            txtUsername.innerHTML = "NAME AVAILABLE!";

            document.getElementById("continue").disabled = false;

            error = false;

        } else if (response === "taken") {
            txtUsername.innerHTML = "NAME TAKEN!";

        } else if (response === "invalid_too_short") {
            txtUsername.innerHTML = "TOO SHORT!";

        } else {
            txtUsername.innerHTML = "AJAX ERROR!";
        }

        // refactor error actions
        if (error) {
            imgUsername.src = img_sad;
            fieldUsername.className = 'error';
        } else {
            imgUsername.src = img_smile;
            fieldUsername.className = 'validated';
        }
}