如何从浏览器中抓取网站？

Question

I would like to scrape a website by just running code in a browser. In this case, the scraper has to run on a specific machine, and I cannot install any software on that machine. However, there is already a browser installed (recent version of Firefox), and I can configure the browser however I want.

What I would like is a javascript solution for scraping, contained in a webpage on site A, that can scrape site B. It seems like this would run into some CORS-type problems; I assume that part of the solution is to disable any cross-origin checks in the browser.

What I have tried so far: I looked up "web scraping in javascript", this brings up a lot of stuff intended to run in nodejs with cheerio for example this tutorial , and also stuff like pjscrape which requires PhantomJS. However, I couldn't find anything equivalent that is intended to run in a browser.

PS This is interesting: Firefox setting to enable cross domain ajax request Apparently Chrome --disable-web-security takes care of the cross-origin/cross-domain issues. Firefox equivalent?

PS Looks like ForceCORS extension to Firefox is also useful: http://www-jo.se/f.pfleger/forcecors I'm not sure if I'll be able to install that though.

PS This is a nice collection of ways to allow cross-domain in different browsers: http://romkey.com/2011/04/23/getting-around-same-origin-policy-in-web-browsers/ Sadly, the suggested Firefox solution doesn't work in versions >=5.

Answer 1

edit: looks like import.io service shut down and the url points to something completely different now. consider this answer obsolete.

try to do it with import.io : ( basically a scraping service with REST API)

as soon as i have a example javascript call to the API i can provide it. Or you check the docs yourself.

Import.io allows you to structure the data you find on webpages into rows and columns, using simple point and click technology.

First you locate your data: navigate to a website using our browser (download it from us here: http://import.io ).

Then, enter our dedicated data extraction workflow by clicking the pink IO button in the top right of the Browser.

We will guide you through structuring the data on the page. You teach import.io how to extract the data by showing us examples of where the data is. We create learning algorithms that generalize from these examples to work out how to get all the data on the website. The data you collect is stored on our cloud servers to be downloaded and shared. And every time you publish to our platform we create an API to get the data programatically so you can easily integrate live web data into your applications or third party analytics and visualization software.

EDIT:

If the data recognition works in the browser you can simply access the data by heading to "simple API integration" and Copy the url

the url u can paste here:

function reqListener () {
    console.log(JSON.parse(this.responseText));
    return JSON.parse(this.responseText);
}

var oReq = new XMLHttpRequest();
oReq.addEventListener("load", reqListener);
oReq.open("GET", "yourUrlFromClipboardComesHere", true);
oReq.send();

xhr request source