将包含unicode文件的内容复制到c中的char数组中

Question

i write a c code as following, that copy a file. it works truely for unicode files (exe, rar for example), i use of a char data-type array to copy file "block" in that. i know that, char data-type just can store 1 byte as extended ASCII standard.

in fread() function, used of buffer[buflen] variable as char array due to copy a block of an exe file ( 100 byte) in that, then copy buffer[buflen] contain in an other file. how it possible that, a block of unicode characters, stores in char ? why this code works for unicode files truely without any problem?

copyFile function :

void copyFile(const char *src, const char *dst)
{
    const int buflen = 100;
    char buffer[buflen];
    long fileSize, curFileSize, offset = 0;
    FILE *r, *w;

    r = fopen(src, "r+b");
    w = fopen(dst, "w+b");

    fseek(r, 0, SEEK_END);
    fileSize = ftell(r);
    fseek(r, 0, SEEK_SET);

    while(fileSize - (curFileSize = ftell(r)) >= buflen)
    {
        fseek(r, offset * buflen, SEEK_SET);
        fread(&buffer, sizeof(buffer), 1, r);
        fwrite(&buffer, sizeof(buffer), 1, w);
        offset++;
    }

    if ((fileSize - curFileSize) != 0)
    {
        fseek(r, (offset - 1) + (curFileSize), SEEK_SET);
        fread(&buffer, fileSize - curFileSize, 1, r);
        fwrite(&buffer, fileSize - curFileSize, 1, w);
    }

    fclose(w);
    fclose(r);
}

entrypoint section :

int main()
{
    copyFile("e:/1.exe", "e:/2.exe");
    return 0;
}

what is the reason of using char data-type or a struct (containing of char ) in fread and fwrite functions?

Thanks of everybody to help me.

Answer 1

Any file, regardless of encoding, is just a sequence of bytes. The char type can store any byte, so you're just copying the file byte for byte. ( char is used in C and C++ as both a character type and a numeric type capable of holding a byte. This can be confusing, but both usages are valid.)

fread and fwrite are specified in terms of char because they read and write bytes.

Answer 2

Well, the file you are reading will probably be encoded with the utf-8 encoding, which makes the utf chars in the range U+0000 --- U+007f the same as their ASCII counterparts (this allows reading normally, even if you don't have a UNICODE compliant reader). The characters in the iso-latin-? set, map normally into two character sequences, and characters like € into three character sequences or more. As long as you don't modify the data you are reading, it doesn't matter the kind of data stored ---begin it binary or textual, or the encoding used---, the copy will be exactly equal than the original (or you will have to look into your code, because it is changing the copy, making it to appear different than the original)

Normally, you will not have any problem, as long as you don't break any of these sequences (which means they came together to the file and you wrote them separately ---to different places--- on the copy) This doesn't happen in a file copy normally. Determining the beginning of a UTF-8 or UTF-16 character is relatively easy, as all characters in a UNICODE encoding can be identified, either going forward or backwards in the data stream.

For UTF-8, the characters are composed of a first character, which encodes the number of bytes on this character, and a tail of n-1 such characters (again, easily detectable) The first character will be 0b110xxxxx ( 0b means the octet in its binary representation from now on) for a two byte character, 0b1110xxxx for a three byte, and so on up to 0b1111110x for a six bytes character) the rest of characters that follow it, are encoded as 0b10xxxxxx . If you go forward, once you se a byte with the MSB set, you know you are in front of a multibyte sequence, you have to count the number of ones on top before the first 0 and you have the number of bytes that compose the character. Going backwards, you first encounter a 0b10xxxxxx char, and you have to go backwards until you get a 0b11xxxxxx char, which will be the first char in a sequence. Then you use the first procedure again.

In UTF-16 the procedure is almost the same. Characters under 0x10000 are encoded as one 16 bit number, and characters equal or above are encoded using a surrogate pair of 16 bit numbers, they have the following pattern: 0b110110xxxxxxxxxx for the first 16bit of the pair, and 0b110111xxxxxxxxxx for the second. This time, you have to substract 0x10000 to the UTF character number before you get the x's that go in the xxxx... part of the two 16bit quantities, but the procedure is similar to the used in utf-8.

In UTF-32 encoding, all the characters are stored as 32bit quantities, so for the moment there's no plan for multisequence encoding. All characters are transmitted as 32bit quantities. At the moment of this writting, the standard is V8.0 and incorporates 1,114,112 code points.

When another UTF encoding is used, for example, UTF-16, all characters are encoded as 16bit quantities, that could change for example if you read them in a little-endian architecture, but you wrote them in a big-endian architecture (you should swap each two bytes for the characters to conserve their UTF values in the target architecture) but again, there can be tricks to cope with this (there's a BOM special signature that allows to check with endianness is being used in the data) so, as long as you copy a file, byte by byte, no reordering of characters is done and the final image is exactly the same as the one you had previously, so UTF should not be concerned.

In variable length encodings ( utf-5 , utf-7 , utf-8 and utf-16) the problem arises if you break one of the multiple sequences that map into the actual UTF codes, because this makes the character non-recognizable by the decoding process (it becomes an illegal character) and then you normally get some special character in the output, signalling the invalid character detected. In constant length encodings (utf-32) you get a broken char only if you split your file at a non multiple of 32bits boundary.

UTF was designed with the aim of being an efficient way to store and send a practically unbound set of characters, and to achieve this, it maps (or tries to map) the most common characters as one byte, augmenting the lenght as more specific or rare characters are selected.

The main source of information about UNICODE is at UNICODE FORUM , where you will find specifications, guidelines and even character maps for the full UNICODE range. UTF-8, UTF-16 and UTF-32 encodings are described here. For utf-5 and utf-7 , you have to follow the above links.