Close MySQL connection upon Python exception within Scrapy Framework?

Question

I am using Scrapy 2.4.x pipeline.py to write data sets to a remote MySQL 5.7.32 server. In some cases errors happen and the script throws an exception - which is OK.

for selector in selectors:
    unit = unitItem()
    try:            
        unit['test'] = selector.xpath('form/text()').extract_first()
        if not unit['test']:
            self.logger.error('Extraction failed on %s', response.url)
            continue    
        else:
            unit['test'] = str(unit['test'].strip()      
    except Exception as e:
        self.logger.error('Exception: %s', e)
        continue

    # more code

    yield unit 

There are 2 problems:

1. RAM usage is climbing up constantly. Do I somehow need to destroy the item?
2. There are many MySQL abborted connections errors. I believe this is due to MySQL connection is not closed

mysql error Log:

Aborted connection 63182018 to db: 'mydb' user: 'test' host: 'myhost' (Got an error reading communication packets)

The connection got opened at the very beginning of process_item and get closed at the very end of the method.

Would it help to close the connection upon exception? If so, is there a recommended routine?

Answer 1

I believe it would be more effective to open SQL connection on spider_opened() and close it on spider_closed()

The only thing to keep in mind is that spider_closed() signal is fired when spider is closed gracefully.