fstream::open() Unicode or Non-Ascii characters don't work (with std::ios::out) on Windows

Question

In a C++ project, I want to open a file ( fstream::open() ) (which seems to be a major problem). The Windows build of my program fails miserably.

• File "ä" (UTF-8 0xC3 0xA4)

   std::string s = ...; //Convert s std::fstream f; f.open(s.c_str(), std::ios::binary | std::ios::in); //Works (f.is_open() == true) f.close(); f.open(s.c_str(), std::ios::binary | std::ios::in | std::ios::out); //Doesn't work

  The string s is UTF-8 encoded, but then converted from UTF-8 to Latin1 (0xE4). I'm using Qt, so QString::fromUtf8(s.c_str()).toLocal8Bit().constData() .

  Why can I open the file for reading, but not for writing?

• File "и" (UTF-8 0xD0 0xB8)

  Same code, doesn't work at all.

It seems, this character doesn't fit in the Windows-1252 charset. How can I open such an fstream (I'm not using MSVC, so no fstream::open(const wchar_t*, ios_base::openmode) )?

Answer 1

Using the standard APIs (such as std::fstream) on Windows you can only open a file if the filename can be encoded using the currently set "ANSI Codepage" (CP_ACP).

This means that there can be files which simply cannot be opened using these APIs on Windows. Unless Microsoft implements support for setting CP_ACP to CP_UTF8 then this cannot be done using Microsoft's CRT or C++ standard library implementation.

(Windows has had a feature called "short" filenames where, when enabled, every file on the drive had an ASCII filename that can be used via standard APIs. However this feature is going away so it does not represent a viable solution.)

Update: Windows 10 has added support for setting the codepage to UTF-8

Answer 2

In Microsoft implementations of STL, there's a non-standard extension (overload) to allow unicode support for UTF-16 encoded strings.

Just pass UTF-16 encoded std::wstring to fstream::open(). This is the only way to make it work with fstream.

You can read more on what I find to be the easiest way to support unicode on windows here: http://utf8everywhere.org/