请求（npm程序包）未返回完整的html

Question

I'm using request to get the HTML of a web page. When I do this for http://orangina.eu/ it only returns some of the HTML. I noticed that it is the same HTML that you get when you use "View Page Source" in Chrome versus the HTML you get when you use "Inspect", which is the full HTML. My guess is that it is getting the HTML before additional HTML is loaded via Javascript. I reviewed the Request documentation and didn't see anything about this.

Why is this happening and is there a way to get the full HTML (using Request or any other package)? Thanks.

Answer 1

Thanks, Andy. Andy answered the question in the comment but I'll add it here so that the question is officially answered and add some more detail that I learned after following Andy's lead. The NPM package Puppeteer solves this problem. It allows you to run a headless Chrome browser within your Node app.

There's one thing that I learned when using Puppeteer to get the http://orangina.eu/ HTML that I want to share. You'll notice that it takes the site a couple of seconds to load. So if you use this code:

const browser = await puppeteer.launch();

const page = await browser.newPage();
await page.goto('http://orangina.eu/');

console.log(await page.content());
await page.screenshot({path: 'screenshot.png'});

await browser.close();

you will get the same thing that I was getting with Request - a small portion of the eventual HTML. This is because they are both grabbing the HTML before it has loaded. Fortunately, Puppeteer has the option to wait before getting the content. I looked to see see if Request has this and did not find anything. Here's the code that gets all of the HTML, notice the 5 second wait:

const browser = await puppeteer.launch();

const page = await browser.newPage();
await page.goto('http://orangina.eu/');
await page.waitFor(5000);

console.log(await page.content());
await page.screenshot({path: 'screenshot.png'});

await browser.close();