How can I scrape this page using Scrapy and Playwright?

Question

I've been on this for a couple of days and couldn't find a solution. My output file comes empty.

Please, help me out

The settings.py file is accordingly to the documentation

Thanks

import scrapy
from scrapy.utils.response import open_in_browser
from scrapy_playwright.page import PageMethod
from scrapy.selector import Selector

class ShopSpider(scrapy.Spider):
    name = 'shop'
    
    def start_requests(self):
        yield scrapy.Request(
            url='https://www.instacart.com/store/sprouts/collections/beef', 
            callback=self.parse, 
            meta={
                'playwright': True, 
                'playwright_page_methods': [
                    PageMethod('wait_for_selector', 'ul.css-qcn8wk-LockupBLarger > li:nth-child(16)')
                ], 
                'playwright_include_page': True
            }, 
            errback=self.close_page
        )


    async def parse(self, response):
        page = response.meta['playwright_page']
        
        for i in range(32, 67, 32):
            await page.evaluate("window.scrollBy(0, document.body.scrollHeight)")
            # count = 16 * i
            await page.wait_for_selector(f'ul.css-qcn8wk-LockupBLarger > li:nth-child({i}) > div > div > div > a > div > div:nth-child(2) > h2 > span')
        s = scrapy.Selector(text=await page.content())
        await page.close()   
        for q in s.css('ul.css-qcn8wk-LockupBLarger > li > div > div > div > a > div > div:nth-child(2) > h2 > span::text').getall():
            yield {
                'title': q
                }

    async def close_page(self, failure):
        page = failure.request.meta["playwright_page"]
        await page.close()
        

Answer 1

Ahh love a bit of a scrape:P....looks like your spider is not generating any output files. One reason for this is that you may not have configured the FEED_URI and FEED_FORMAT settings in your settings.py file, or you might have the wrong file path.

FEED_URI = 'output/items.json'
FEED_FORMAT = 'json'

Also, another thing to be aware of is that, you must use the yield keyword instead of return inside the parse method, to return the data to the spider and storing it to the output file.

Hope that helps a bit