C＃对UTF-16字节数组执行字符串操作

Question

I'm reading a file into byte[] buffer . The file contains a lot of UTF-16 strings (millions) in the following format:

• The first byte contain and string length in chars (range 0 .. 255)
• The following bytes contains the string characters in UTF-16 encoding (each char represented by 2 bytes, means byteCount = charCount * 2).

I need to perform standard string operations for all strings in the file, for example: IndexOf , EndsWith and StartsWith , with StringComparison.OrdinalIgnoreCase and StringComparison.Ordinal .

For now my code first converting each string from byte array to System.String type. I found the following code to be the most efficient to do so:

// position/length validation removed to minimize the code

string result;
byte charLength = _buffer[_bufferI++];
int byteLength = charLength * 2;

fixed (byte* pBuffer = &_buffer[_bufferI])
{
    result = new string((char*)pBuffer, 0, charLength);
}

_bufferI += byteLength;
return result;

Still, new string(char*, int, int) it's very slow because it performing unnecessary copying for each string .

Profiler says its System.String.wstrcpy(char*,char*,int32) performing slow.

I need a way to perform string operations without copying bytes for each string .

Is there a way to perform string operations on byte array directly ?

Is there a way to create new string without copying its bytes?

Answer 1

No, you can't create a string without copying the character data.

The String object stores the meta data for the string (Length, et.c.) in the same memory area as the character data, so you can't keep the character data in the byte array and pretend that it's a String object.

You could try other ways of constructing the string from the byte data, and see if any of them has less overhead, like Encoding.UTF16.GetString .

If you are using a pointer, you could try to get multiple strings at a time, so that you don't have to fix the buffer for each string.

Answer 2

You could read the File using a StreamReader using Encoding.UTF16 so you do not have the "byte overhead" in between:

using (StreamReader sr = new StreamReader(filename, Encoding.UTF16)) 
{
    string line;

    while ((line = sr.ReadLine()) != null) 
    {
        //Your Code
    }
}

Answer 3

You could create extension methods on byte arrays to handle most of those string operations directly on the byte array and avoid the cost of converting. Not sure what all string operations you perform, so not sure if all of them could be accomplished this way.