Python沙皮蜘蛛

Question

I want to scrape data from a website http://www.quoka.de/immobilien/bueros-gewerbeflaechen with this filter:

<a class="t-bld" rel="nofollow" href="javascript:qsn.set('classtype','of',1);">nur Angebote</a>

How to set this filter using scrapy?

Answer 1

You can parse a specific website by using Beautifulsoup and urllib2 . Here is a python implementation for the data that you wanted to parse or scrape according to the filter you wrote.

from BeautifulSoup import BeautifulSoup
import urllib2

def main1(website):
    data_list = []
    web =urllib2.urlopen(website).read()
    soup = BeautifulSoup(web)
    description = soup.findAll('a', attrs={'rel':'nofollow'})
    for de in description:
        data_list.append(de.text)
    return data_list

print main1("http://www.quoka.de/immobilien/bueros-gewerbeflaechen")

If you wanted to parse other data such as the description from the following:

def main(website):
    data_list = []
    web =urllib2.urlopen(website).read()
    soup = BeautifulSoup(web)
    description = soup.findAll('div', attrs={'class':'description'})
    for de in description:
        data_list.append(de.text)
    return data_list

print main("http://www.quoka.de/immobilien/bueros-gewerbeflaechen") #this is the data of each section

Answer 2

One way is by submitting a request with the parameter and parse the result of the response. See the following code sample:

import scrapy

class TestSpider(scrapy.Spider):

    name = 'quoka'
    start_urls = ['http://www.quoka.de/immobilien/bueros-gewerbeflaechen']

    def parse(self, response):

        request = scrapy.FormRequest.from_response(
            response,
            formname='frmSearch',
            formdata={'classtype': 'of'},
            callback=self.parse_filtered
        )
        # print request.body
        yield request

    def parse_filtered(self,response):

        searchResults = response.xpath('//div[@id="ResultListData"]/ul/li')
        for result in searchResults:
            title = result.xpath('.//div[@class="q-col n2"]/a/@title').extract()
            print title