It's the String conversion again: UNIX Windows-1252 to String

Question

I'm downloading a website in Java, using all this:

myUrl = new URL("here is my URL");
            in = new BufferedReader(new InputStreamReader(myUrl.openStream()));

In this file however there are some special characters like ä,ö and ü. I need to be able to print these out properly.

I try to encode the Strings using:

String encodedString = new String(toEncode.getBytes("Windows-1252"), "UTF-8");

But all it does is replace these special characters with a ? .

When I open what I am trying to print here using a downloaded .html file from Chrome with Notepad++, it says (in the bottom right corner) UNIX and Windows-1252 . That's all I know about the encoded file.

What more steps can I take to figure out what is wrong?

--AND--

How can I convert this file so that I can properly read and print it in Java?

Sorry if this question is kind of stupid... I simply don't know any better and couldn't find anything on the internet.

Answer 1

OK, so you are mixing a lot of things here.

First of all, you do:

new InputStreamReader(myUrl.openStream())

this wil open a reader, yes; however, it will do so using your default JRE/OS Charset . Maybe not what you want.

Try and specify that you want UTF_8 (note, Java 7+ code):

try (
    final InputStream in = myUrl.openStream();
    final Reader reader = new InputStreamReader(in, StandardCharsets.UTF_8);
) {
    // read from the reader here
}

---

Now, what you are mixing...

You read from an InputStream ; an InputStream only knows how to read byte s.

But you want text; and in Java, text means a sequence of char s.

Let us forget for a moment that you want char s and focus on the fact that you want text; let us substitute a char for a carrier pigeon.

Now, what you need to do is to transform this stream of bytes into a stream of carrier pigeons. For this, you need a particular process. And in this case, the process is called decoding .

Back to Java, now. There also exists a process which does the reverse: encoding a stream of carrier pigeons (or char s) into a stream of bytes.

The trick... There exist several ways to do that; Unicode refers to them as character codings ; and in Java, the base class which provides both encoders and decoders is a Charset .

Now, an InputStreamReader accepts a Charset as a second argument... Which you should ALWAYS specify. If you DO NOT, this:

new InputStreamReader(in);

will be equivalent to:

new InputStreamReader(in, Charset.defaultCharset());

and Charset.defaultCharset() is Not. Guaranteed. To. Be. The. Same. Amongst. Implementations. Of. JREs.