Sending custom cookies (and viewing the ones set naturally) with Perl's WWW::Mechanize

Question

I'm trying to scrape a particularly troublesome website. Though all the parameters match and the referrer matches, I see different results when perl runs it than when I watch dev tools.

When I do a copy-as-curl from dev tools, the only header I can't confirm as identical is -H 'Cookie: and its contents. Running that curl command gives me the proper results just as I receive in the browser.

So, what syntax do I use with WWW::Mechanize to set the cookie's value explicitly rather than letting Mechanize do it for me based on the past gets/posts?

Also, how can I view what it does want to set the cookie's value to?

Answer 1

To examine the cookies returned from a WWW::Mechanize request, use the following:

my $cookie_jar = $mech->cookie_jar; # returns a HTTP::Cookies object
print $cookie_jar->as_string, "\n”;

To set a cookie for use by WWW::Mechanize in requests, you would do the following:

$mech->cookie_jar->set_cookie(-name=>'YourCookieName',
                   -value=>'YourCookieValue',
                   -host=>'www.your website.com',
                   -expires=>'Sun, 31 Jan 2021 18:45:47 GMT',
                   -path=>'/'
                   -secure=>'false');

Refer to the HTTP::Cookies documentation for other useful methods.