[英]Fast Data insertion in SQL Server using python
I want to insert data from a CSV file into SQL Server database hosted on Azure.我想将 CSV 文件中的数据插入到托管在 Azure 上的 SQL 服务器数据库中。 I was able to insert data in the table by reading the data into a pandas dataframe and using insert statement in a for loop in python.通过将数据读入 pandas dataframe 并在 python 的 for 循环中使用 insert 语句,我能够在表中插入数据。 I am using pyodbc
.我正在使用pyodbc
。 This approach took a long time for the data to get inserted.这种方法需要很长时间才能插入数据。 I also tried pd.to_sql()
.我也试过pd.to_sql()
。 Though the later is faster than for loop approach, it is still slow.虽然后者比 for 循环方法快,但它仍然很慢。
Is there any faster way to insert CSV file in SQL Server using python/pandas?有没有更快的方法使用 python/pandas 在 SQL 服务器中插入 CSV 文件?
Use threads so each one can insert into the db.使用线程,以便每个线程都可以插入数据库。 A good example is provided by this guy who has a great example.这个人提供了一个很好的例子,他有一个很好的例子。 Check this link .检查此链接。
Take a look at this part of the code where he starts the threads that point to the insert function.看看这部分代码,他启动了指向插入 function 的线程。
def rnd_user(num=1000001, threadid=1):
query = u"INSERT INTO imdb.employees (fname, lname, hired, job_code, store_id) VALUES ('%(fname)s','%(lname)s','%(hired)s','%(jobcode)s','%(storeid)s');"
cnx = mysql.connector.connect(**dbconfig)
cnx.autocommit = True
cursor = cnx.cursor()
def rnd_date():
return time.strftime("%Y-%m-%d", (random.randrange(2000,2016), random.randrange(1,12), random.randrange(1,28), 0, 0, 0, 0, 1, -1))
for x in range(num):
if not shutdown_event.is_set():
fname = genstring(3, 9)
lname = genstring(4, 12)
hired = rnd_date()
jobcode = genstring(3, 3).upper()
storeid = random.randrange(1, 20)
cursor.execute(query % {u'fname': fname, u'lname': lname, u'hired': hired, u'jobcode': jobcode, u'storeid': storeid})
if x % 1000 == 0:
print "[%2d] Inserted %d rows" % (threadid, x)
cnx.close()
... (more code) ...
for x in range(8):
t = threading.Thread(target=rnd_user, args=(125000,threadId,))
t.start()
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.