Yahoo Finance API Java下载CSV

Question

I have been using the Yahoo Finance "API" with the code below for a few weeks. Since about a week ago, it stopped working. If you type in this address: http://ichart.yahoo.com/table.csv?s=MSFT , you'll get some historical data for Microsoft (my browser automatically downloads it). However, when I try to read it through Java, the stream is apparently opened (I don't get an exception), however, the stream contains no data. Anyone know why the buffered reader below is not able to stream the object as it had been able to do in previous weeks? I suspect that maybe Yahoo added some java script to block automated downloading.

URL url = new URL("http://ichart.yahoo.com/table.csv?s=" + symbol);
URLConnection hc = url.openConnection();
hc.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 4.10; rv:52.0) Gecko/20100101 Firefox/52.0");
BufferedReader in = new BufferedReader(new InputStreamReader(hc.getInputStream()));
    br.readLine();
    //Read File Line By Line
    String strLine;
    while ((strLine = br.readLine()) != null) {
        System.out.println(strLine);
    }
    br.close();

Answer 1

Many Websites are checking for a User Agent String and block acess if no String has been sent to block automated access.

This is an example how you can add the user agent string:

    URL url = new URL("http://ichart.yahoo.com/table.csv?s=MSFT");
    URLConnection hc = url.openConnection();
    hc.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 4.10; rv:52.0) Gecko/20100101 Firefox/52.0");

Answer 2

To understand the reason the java communication doesn't work, track the html communication. I used Chrome browser, with HTTP trace extension enabled. The trace output shows that the link http://ichart.yahoo.com/table.csv?s= ^GSPC redirects to https://ichart.yahoo.com/table.csv?s= ^GSPC.

In Java, you need to implement a URL connection to follow the redirect. An example of following the redirect is at http://www.mkyong.com/java/java-httpurlconnection-follow-redirect-example/ . Modify the example by passing your url string, and you should get the expected output.

Answer 3

您可以将URL从“ http”更改为“ https”，这样可以正常工作。

Answer 4

Try executing this code from some other network. That'll give you clarity whether Yahoo has blocked automated downloading or only blacklisted the network you're using.