I am still new to Scrapy and Python.
I was able to connect on my remote DB couple with SSH credentials but...
would like to prevent this error from happening on every item being scraped.
Error: 2055: Lost connection to MySQL server at '127.0.0.1:3306', system error: 10053 An established connection was aborted by the software in your host machine
Below is my MySQL Pipeline Object
import mysql.connector
import sshtunnel
class MySQLStorePipeline(object):
def __init__(self):
with sshtunnel.SSHTunnelForwarder(
('13.***.***.***', 22),
ssh_username='***',
ssh_password='***',
remote_bind_address=('db1.prod.***.***.net.***', 3306),
local_bind_address=('127.0.0.1', 3306)
) as tunnel:
self.dbcon = mysql.connector.connect(
host='127.0.0.1',
port=3306,
user='***',
database='***',
password='***',
charset='utf8'
)
self.cursor = self.dbcon.cursor()
def process_item(self, item, spider):
try:
tables = self.cursor.execute('SHOW TABLES')
print tables.fetchall()
self.dbcon.commit()
except mysql.connector.Error as err:
print('Error: {}'.format(err))
return item
I just don't know how to maintain a database connection inside process_item function
You're using with ...
that's why you get this behavior SSH tunnel from Python is closing automatically
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.