如何将任意二进制数据插入VARCHAR列？

Question

I have a MySQL table with a VARCHAR(100) column, using the utf8_general_ci collation.

I can see rows where this column contains arbitrary byte sequences (ie data that contains invalid UTF8 character sequences), but I can't figure out how to write an UPDATE or INSERT statement that allows this type of data to be entered.

For example, I've tried the following:

UPDATE DataTable SET Data = CAST(BINARY(X'16d7a4fca7442dda3ad93c9a726597e4') AS CHAR(100)) WHERE Id = 1;

But I get the error:

Incorrect string value: '\xFC\xA7D-\xDA:...' for column 'Data' at row 1

How can I write an INSERT or UPDATE statement that bypasses the destination column's collation, allowing me to insert arbitrary byte sequences?

Answer 1

Have you considered using one of the Blob data types instead of varchar? I believe that this'd take a lot of the pain away from your use-case.

EDIT: Alternatively, there is the HEX and UNHEX functions, which MySQL supports. Hex takes either a str or a numeric argument and returns the hexadecimal representation of your argument as a string. Unhex does the inverse; taking a hexadecimal string and returning a binary string.

Answer 2

The short answer is that it shouldn't be possible to insert values with invalid UTF8 characters into VARCHAR column declared to use UTF8 characterset.

That's the design goal of MySQL, to disallow invalid values. When there's an attempt to do that, MySQL will return either an error or a warning, or (more leniently?) silently truncate the supplied value at the first invalid character encountered.

The more usual variety of characterset issues are with MySQL performing a characterset conversion when a characterset conversion isn't required.

But the issue you are reporting is that invalid characters were inserted into a UTF8 column. It's as if a latin1 (ISO-8859) encoding was supplied, and a characterset conversion was required, but was not performed.

As far as working around that... I believe it was possible in earlier versions of MySQL. I believe it was possible to cast a value to BINARY, and then warp that in CONVERT( ... USING UTF8) , and MySQL wouldn't perform a validation of the characterset. I don't know if that's still possible with the current MySQL Connectors.

If it is possible, then that's (IMO) a bug in the Connector.

The only way I can think of getting around that characterset check/validation would be to get the MySQL sever to trust the client, and determine that no check of the characterset is required. (That would also mean the MySQL server wouldn't be doing a characterset conversion, the client lying to the server, the client telling the server that it's supplying valid UTF8 characters.

Basically, the client would be telling the server "Hey server, I'm going to be sending UTF8 character encodings".

And the server says "Okay. I'll not do any characterset conversion then, since we match. And I'll just trust that what you send is valid UTF8".

And then the client mischievously chuckles to itself, "Heh, heh, I lied. I'm actually sending character encodings that aren't valid UTF8".

And I think it's much more likely to be able to achieve such mischief using prepared statements with the old school MySQL C API ( mysql_stmt_prepare , mysql_stmt_execute ), supplying nvalid UTF8 encodings as values for string bind parameters. (The onus is really on the client to supply valid values for bind parameters.)

Answer 3

您应该预先对值进行base64编码，以便可以使用它生成有效的SQL：

UPDATE DataTable SET Data = from_base64('mybase64-encoded-representation-of-my-value') WHERE Id = 1;