'Set-Cookie' is not included in the response headers

Question

I'm trying to get the set-cookie header from the http response, but it's not showing up for most of the requests.

Using https://www.southwest.com/ as an example, you can see that https://bs.serving-sys.com/Serving/ActivityServer.bs?cn=as&ActivityID=1345510&rnd=459203.51759912557&fare%20class=[fare%20class]&business%20or%20leisure=[business%20or%20leisure]&number%20of%20passengers=[number%20of%20passengers]&date=[date]&destination=[destination]&origination=[origination] sets 3 cookies:

Puppeteer code:

const puppeteer = require('puppeteer');

async function getResponseCookies() {
    function handleResponse(response) {
        const url = response.url();
        const headers = response.headers();
        const status = response.status()

        if(url.includes('https://bs.serving-sys.com/Serving/ActivityServer.bs')) {
            console.log('RESPONSE URL ', url)
            console.log('RESPONSE HEADERS ', headers)
            console.log('RESPONSE STATUS ', status)
        }
    }

    const browser = await puppeteer.launch({
        ignoreDefaultArgs: ["--enable-automation"],
        executablePath: "/usr/bin/google-chrome",
        headless: true,
        ignoreHTTPSErrors: true,
    });

    const page = await browser.newPage();

    await page.setUserAgent("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36");

    await page.on('response', async(response) => {
        await handleResponse(response);
    })

    const urls = ['https://www.southwest.com'];

    for(let url of urls) {
        await page.goto(url, { timeout: 0, waitUntil: 'networkidle0' });
    }
    await browser.close();
}

getResponseCookies();

The above code execution outputs the following log, without any 'set-cookie' headers in the response:

RESPONSE URL  https://bs.serving-sys.com/Serving/ActivityServer.bs?cn=as&ActivityID=1345510&rnd=68456.37277058625&fare%20class=[fare%20class]&business%20or%20leisure=[business%20or%20leisure]&number%20of%20passengers=[number%20of%20passengers]&date=[date]&destination=[destination]&origination=[origination]
RESPONSE HEADERS  { pragma: 'no-cache',
  date: 'Mon, 03 Feb 2020 10:30:16 GMT',
  'content-encoding': 'gzip',
  server: 'Microsoft-IIS/7.5',
  'x-powered-by': 'ASP.NET',
  p3p: 'CP="NOI DEVa OUR BUS UNI"',
  'access-control-allow-origin': '*',
  'cache-control': 'no-cache, no-store',
  'content-type': 'text/html; charset=UTF-8',
  'content-length': '616',
  expires: 'Sun, 05-Jun-2005 22:00:00 GMT' }

Any ideas why the Set-Cookie header is missing from the response?

*Note that those cookies are returned when using Network.getAllCookies from CDP

Answer 1

According to https://github.com/puppeteer/puppeteer/issues/4918 , it looks like puppeteer is not listening to the Network.responseReceivedExtraInfo event that contains the raw headers. Listening to that event did the trick for me.