NodeJS | 使用请求模块等待页面重定向

Question

When accessing http://www.oneblockdown.it/en/pharrell-williams-x-adidas-originals/footwear-lifestyle/men-unisex/pharrell-williams-x-adidas-originals-human-race-nmd-bb3070/4267 through a normal browser, it will often sit on a white page for 2.5 seconds, run some javascript, then automatically refresh the page and then it will load the actual product page.

When I request the page using Node JS Request module (which uses the normal NodeJS http/https modules) it returns the html of the blank white page instead of waiting and showing the product page. The html of the blank looks as such: https://pastebin.com/ZSeXuZRc

How would I go about allowing it to wait, execute the code as if its a browser and allowing it to refresh and show the product page?

Here is my current code:

    request(
    {
        url: 'https://www.oneblockdown.it/en/pharrell-williams-x-adidas-originals/footwear-lifestyle/men-unisex/pharrell-williams-x-adidas-originals-human-race-nmd-bb3070/4267',
        method: 'GET',
        headers: {
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36',
            'Host': 'www.oneblockdown.it',
            'Cache-Control': 'max-age=0',
            'Upgrade-Insecure-Requests': '1',
            'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
            'Referer': 'http://www.oneblockdown.it/en/footwear',
            'Accept-Encoding': 'gzip, deflate',
            'Accept-Language': 'en-US,en;q=0.9,la;q=0.8'

        }
    },
    function(err, res, body) {
        if(err || res.statusCode !== 200){
            console.log(`FAILED | ${err}`)
        } else {
          //parse the body here. It should return product page, but is returning the white html with javascript
        }
    });

Answer 1

Unfortunately, you cannot do so. The response includes everything that the server returns when you send a request to that endpoint. If there is javascript that gets invoked upon the DOM loading ie window.onload or document.ready which is the case for this webpage, you will not get those changes.

This would be a problem even for a web crawler (that's why there are search engine optimization standards related to serving all important information without the use of any scripting and leaving UI related tasks such as styling and interactions once the page has loaded.