用Javascript抓取网页？

Question

I'm looking for a way to get all the sentences in 'lead'-tags from this webpage: http://taz.de/!p4633/c.xml and put them into an array. Can a javascript program get information from the web like that?

for example, where it says

<lead>Sentence1 blablabla. sentence2 bla bla bla.</lead>
<headline>something else</headline>
<lead>sentence3 blablabla. sentence4 bla bla.</lead>

I'd like to get the strings like so:

var sentences = ["Sentence1 blablabla.", "sentence2 bla bla bla.", "sentence3 blablabla.", "sentence4 bla bla."];

The reason is that I want to make a twitterbot that answers with random sentences from this newspaper's website. I searched tutorials for webscraping, but I'm not familiar with node.js and couldn't get any of the other tools to work either, because I know so little about programming.

Answer 1

Can a javascript program get information from the web like that?

Yes.

You'll need to know about Node's HTTP module , particularly http.get . Then you'll need an XML parser. There should be a bunch floating around in npm, choose any one. Get the XML, parse the XML, pick the pieces of data you want and put them in an array.

Answer 2

This will work for you, I couln't get a real response from your webpage cause of firewall, but try this example code, and tell us:

 var fakeResponse = "<xml><lead>Sentence1 blablabla. sentence2 bla bla bla.</lead></xml>"; function processResponse(response) { var parser = new DOMParser(); var xmlDoc = parser.parseFromString(response, "text/xml"); //important to use "text/xml" for (var i=0; i < xmlDoc.getElementsByTagName("lead").length; i++) { var html = xmlDoc.getElementsByTagName("lead")[i].innerHTML; console.log("item " + i + "=>" + html); } } var xhttp = new XMLHttpRequest(); xhttp.onreadystatechange = function() { if (this.readyState == 4 && this.status == 200) { var responseData = this.responseText; responseData = fakeResponse; //delete This line, just for testing processResponse(responseData) } }; var your_url = "https://jsonplaceholder.typicode.com/posts/1"; //update with the url of your webservice xhttp.open("GET", your_url, true); xhttp.send();