Yahoo finance cookie and crumb not working

Question

I am trying to scrape the cookie and crumb from Yahoo finance in Java to get daily stock price histories.

I modified code from the post Yahoo Finance URL not working , but it does not seem to be working. In particular, when searching for the crumb, the word "CrumbStore" does not seem to appear in the search text anywhere:

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.URL;
import java.net.URLConnection;
import java.util.List;
import java.util.Map;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

public class GetYahooQuotes {

    public static void main(String args[]) throws IOException {

        String cookie = null;

    try {
            // Open the URL connection
        URL url = new URL("https://finance.yahoo.com/quote/SPY");
        URLConnection con = url.openConnection();

        for (Map.Entry<String, List<String>> entry : con.getHeaderFields().entrySet()) {
            if (entry.getKey() == null || !entry.getKey().equals("Set-Cookie"))
                    continue;
                for (String s : entry.getValue()) {
                    // store your cookie
                    cookie = s;
                    System.out.println( "Cookie = " + cookie);
                }
            }

        String crumb = null;
        InputStream inStream = con.getInputStream();
        InputStreamReader irdr = new InputStreamReader(inStream);
        BufferedReader rsv = new BufferedReader(irdr);

        Pattern crumbPattern = Pattern.compile(".*\"CrumbStore\":\\{\"crumb\":\"([^\"]+)\"\\}.*");

        String line = null;
        while (crumb == null && (line = rsv.readLine()) != null) {
            Matcher matcher = crumbPattern.matcher(line);
            if (matcher.matches()) {
                crumb = matcher.group(1);
                System.out.println( "Crumb = " + crumb);
            }
        }
        rsv.close();
    }
        catch (java.net.SocketTimeoutException e) {
        // The URL connection timed out.  Try again.
        }
    }
}

Also, when storing the cookie, it seems like there are three different cookies. Which one should I use? My output is:

Cookie = A1S=d=AQABBLeLr2MCEKfVotZCLRRvj9tdxyruiIkFEgEBAQHdsGO5YwAAAAAA_eMAAA&S=AQAAAtgfY5LB0heCm1MP9BNfRAA&j=WORLD; Domain=.yahoo.com; Path=/; SameSite=Lax; Secure Cookie = A3=d=AQABBLeLr2MCEKfVotZCLRRvj9tdxyruiIkFEgEBAQHdsGO5YwAAAAAA_eMAAA&S=AQAAAtgfY5LB0heCm1MP9BNfRAA; Expires=Sun, 31 Dec 2023 07:09:12 GMT; Max-Age=31557600; Domain=.yahoo.com; Path=/; SameSite=None; Secure; HttpOnly Cookie = A1=d=AQABBLeLr2MCEKfVotZCLRRvj9tdxyruiIkFEgEBAQHdsGO5YwAAAAAA_eMAAA&S=AQAAAtgfY5LB0heCm1MP9BNfRAA; Expires=Sun, 31 Dec 2023 07:09:12 GMT; Max-Age=31557600; Domain=.yahoo.com; Path=/; SameSite=Lax; Secure; HttpOnly

I understand that when I get the crumb, I will have to call something like:

String quoteUrl = "https://query1.finance.yahoo.com/v7/finance/download/IBM?period1=1493425217&period2=1496017217&interval=1d&events=history&crumb="
                           + crumb

but I am not there yet. I still need help on how to get the crumb.

Answer 1

The secure cookie is the cookie I would use out of the three, that way you have a better bet on avoiding 401 unauthorized errors. I also wouldn't doubt if the "CrumbStore" property has been renamed or even removed, as the post you referenced dates back to 2017. I would try some cURL requests or use Postman to try to find what the exact parameter names are.

I found a snippet that also utilizes java.net.UrlConnection and instead of trying to find some type of regular expression pattern, it just searches through the headers of the request. It retrieves the cookie in the process as well.

String headerName = null;
String crumb = null;
for (int i = 1; (headerName = con.getHeaderFieldKey(i)) != null; i++) {
    if (headerName.equals("Set-Cookie")) {                  
        String cookie = con.getHeaderField(i);
        cookie = cookie.substring(0, cookie.indexOf(";"));
        crumb = cookie.substring(cookie.indexOf("=") + 1, cookie.indexOf("&"));
    }
}

And perhaps you can integrate it into your existing code like this:

for (Map.Entry<String, List<String>> entry : con.getHeaderFields().entrySet()) {
    if (entry.getKey() == null || !entry.getKey().equals("Set-Cookie"))
        continue;
        for (String s : entry.getValue()) {
            // store your cookie
            cookie = s;
            System.out.println( "Cookie = " + cookie);
            String crumb = cookie.substring(cookie.indexOf("=") + 1, cookie.indexOf("&"));
            System.out.println( "Crumb = " + crumb);
        }
     }
}