节点js-尝试发送POST请求，但未加载javascript内容

Question

I have been trying to send a POST request to a website, and it has not been loading the full site.

I am guessing that this is a problem with node js not loading the javascript content, then executing it. I can SEE the javascript in the response, though it never loads the content that the javascript files are producing in the html.

What I mean is, it doesn't actually execute the javascript. How would I make node js code to overcome this, and wait for the the full website to load? I have tried native code, and the "request' module, both producing the same errors. If it helps, I don't need native javascript. Any modules would be fine.

Some code I have tried (request module):

var options = {
method: 'POST',
uri: 'https://url',
formData: {
    'email': 'Email',
    'prize': '0',
    'transactional': 'on'
},
headers: {
    /* 'content-type': 'application/x-www-form-urlencoded' */ // Is set automatically
}
};

rp(options)
.then(function (body) {
   console.log(body)
})
.catch(function (err) {
   console.log(err)
});

Answer 1

When you fetch the contents of a URL or POST and get a response, that response is just some text. node.js does not parse the content. It does not execute Javascript in that content. That's what a browser does - node.js is not a browser. So, what you get is the RAW content that would be downloaded to a browser form that website. If it's an HTML response, then you just get the raw HTML page and no embedded scripts are fetched or run.

If you want that page parsed and scripts run in it, then you need to get a component you can use inside of node.js that mimics what a browser does. It's not too hard to get an HTML parser, but it's a lot more work to actually fetch and run the Javascript in that page and do that safely in a way that does not put your server at risk. In fact, Google generally doesn't even run your Javascript when indexing web pages (though they have been working on that technology for some circumstances).

jsDOM and cheerio are a couple modules you can use in node.js for parsing the HTML from a response, jsDOM will execute scripts in the page, cheerio will not.

Another package you could use is phantomJS which can serve as a webkit-based, headless browser which you can communicate with from node.js.