如何使用Scrapy从表中使用ItemLoader抓取数据？

Question

I am trying to extract data from the website " https://www.brickworkratings.com/CreditRatings.aspx ". There is a table through which I can easily extract data through Scrapy Shell.

I wanted to use ItemLoaders as it is really powerful and gives a cleaner experience.

Here is my code below.

def start_requests(self):
    yield Request("https://www.brickworkratings.com/CreditRatings.aspx", self.parse_credit_rating_response)

def parse_credit_rating_response(self, response):
    table_rows = response.xpath('//*[@id="ContentPlaceHolder1_gvData"]//tr')
    for table_row in table_rows:
        loader = ItemLoader(SampleItem(), response=response)
        try:
            loader.get_xpath(table_row.xpath("td[1]//a/text()")[0].extract())
            # loader.add_value('company_name', 'test')
        except Exception as e:
            print(e)
        item = loader.load_item()
        print(item)
        yield item

I am getting the error,

"XPath error: Invalid expression in 
                                                                        (Name of the Company)".

I believe my XPath is right but I don't think this is the way to use it. How do I use it correctly? I need to extract data from the table and wanted to use the more powerful ItemLoaders.

Any help will be appreciated, been stuck on it for a long time.

Answer 1

You need to specify the initial/parent selector when constructing the loader. It is then unnecessary to provide the response as well. And then you need to pass an XPath string to add_xpath instead of using get_xpath . Refer to the documentation .

Assuming your XPath is correct, here is an example:

# All added selectors will now be relative to table_row.
loader = ItemLoader(SampleItem(), selector=table_row)
# Just give it the XPath here.
loader.add_xpath("field_name", "td[1]//a/text()")

If you need to do additional processing, look at input/output processors .