使用 Node.js 和 Puppeteer 进行定时重定向后的 HTTP 响应标头

Question

How to get the response headers using Puppeteer has already been answered below:

Possible to get HTTP response headers with Nodejs and Puppeteer

However, I have a peculiar situation where the initial URL redirects to another URL after a few seconds.

Here is the pertinent code I'm running:

const browser = await puppeteer.launch({args: ['--no-sandbox', '--disable-setuid-sandbox'], headless: false});

const page = await browser.newPage();

// get the response object of the initial URL
var page_response_obj = await page.goto(url_str, {timeout: PAGE_TIMEOUT_GOTO_MS, waitUntil: 'domcontentloaded'});

// get page title of initial page
var page_title_1_str = await page.title();

// wait for a few seconds to cover the timed redirect
await page.waitFor(6130);

// get page title of final page
var page_title_2_str = await page.title();

I can get the page titles of the two different pages, but I'm not sure how to get the response headers given that the page_response_obj will contain the response headers for the initial URL.

Is it possible to get the response headers of the final URL?

EDIT

I'm using this for websites that have CloudFlare protection where you need to wait for about 5 seconds before you get redirected to the actual website.

Answer 1

You can use the chained redirect property of Request object.

const puppeteer = require ('puppeteer')
const url = 'http://doodle.google.com/'

;(async () => {
    const browser = await puppeteer.launch({
        args: ['--no-sandbox', '--disable-setuid-sandbox'],
        headless: true
    })

    const page = (await browser.pages())[0]

    // get the response object of the initial URL
    const response = await page.goto(url, {timeout: 0, waitUntil: 'domcontentloaded'})

    // get the first response header
    console.log ( response.headers() )

    // get page title of initial page
    const title1 = await page.title()

    const chain = response.request().redirectChain()

    // If the page redirected, all of chained response headers will be shown here
    for ( let num in chain ) {
        console.log( chain[num].response().headers() )
        // console.log(chain[0].url()) // => print the URL
    }

    // get page title of final page
    const title2 = await page.title()
})()

Answer 2

On closer inspection, it turns out that there are some redirects that might be coerced on the front-end [via a script], and so may not be captured in a standard redirect chain. As such, I didn't have success with Edi's suggestion.

So here's what I needed to change to get things to work:

1. Use a response event handler
2. Wait for a long while (30 to 45 seconds) to make sure that you capture the relevant response after redirection. You can adjust the length of time if you need to.

In my case, I was trying to determine if gzip is enabled, so I needed a valid response object on the final URL. Here's the revised code:

// define url and host
var url_str = 'https://www.example.com';
var url_host_str = 'example.com';

// define GZIP test function
var _checkGZIP = function(resp_headers_obj)
{
    var resp_header_content_encoding_str = resp_headers_obj['content-encoding'];
    var is_gzip_bool = !!(/gzip/i.test(resp_header_content_encoding_str));

    return is_gzip_bool;
};

const browser = await puppeteer.launch({args: ['--no-sandbox', '--disable-setuid-sandbox'], headless: true});

const page = await browser.newPage();

// set result variable outside event handler scope
var is_gzip_bool = false;

/**
 * Set response event handler
 * This will capture all responses from the initial URL and from final URL
 */
page.on('response', function(response_obj)
{    
    // get URL and headers
    var resp_url_str = response.url();
    var resp_headers_obj = response.headers();

    if(!is_gzip_bool)
    {
        // check for only specific URLs
        if(/^ *https?\:\/\/([^\?\/]+)(\/|)([^\n\r\?\.]+|) *$/i.test(resp_url_str) && resp_url_str.includes(url_host_str))
        {
            // do gzip test
            is_gzip_bool = _checkGZIP(resp_headers_obj);
        }
    }
}

// go to page
await page.goto(url_str, {timeout: PAGE_TIMEOUT_GOTO_MS, waitUntil: 'domcontentloaded'});

// wait for a long while to capture all relevant responses [from both initial and final URL]
await page.waitFor(30000);

// document your result if required

// close browser
await browser.close();