[英]append list of tuples to data frame python panda
tlist = [("a1", "a2","a3"),("b1", "b2","b3"),("c1", "c2","c3")]
我想要的是:
df=pd.DataFrame([["a1","a2","a3"],["b1","b2","b3"],["c1","c2","c3"]])
我可以做這個:
df2 = pd.DataFrame(tlist, columns=['col1', 'col2', 'col3'])
然而,元組列表是從某個數據庫中提取的,所以我有一個循環,並一次提取一個數據塊,然后追加。
什么是最好的方法來做到這一點?
數據拉取現在可以達到 10 億行,並且可以增長。
謝謝。
#very big table#
sql2 = "Select col1,col2,col3 from bigT"
#very big table#
try:
cursor.execute (sql2)
except cx_Oracle.DatabaseError:
print ('Failed \n'+sql2)
#need to do it in chunk as not enough memory and blow up!
while True:
tlist = cursor.fetchmany()
print(type(tlist))
print (len(tlist))
if rows == []:
break;
#I cannot get this one to work
df.append([tlist],ignore_index=True)
#I cannot get this one to work
我找到了一個更簡單的解決方案!
import pandas as pd
print(con.version)
query = """select * from all_tab_columns"""
df_ora = pd.read_sql(query, con=con)
import cx_Oracle
conn_str="scott/tiger@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=db1.org.waaw.com)(PORT=1234))(CONNECT_DATA=(SERVICE_NAME=hatsx)))"
con = cx_Oracle.connect(conn_str)
cursor= con.cursor ()
cursor.arraysize = 10000
import pandas as pd
rowsx=[("xxxxxxxxxx","xxxxxxxxxx","xxxxxxxxxx","xxxxxxxxxx","xxxxxxxxxx")]
labels=['col1', 'col2','col3']
df = pd.DataFrame(rowsx, columns=labels)
#verify the connection
print (con.version)
#verify the connection
#very big table#
sql2 = """Select col1,col2,col3 from bigT"""
#very big table#
try:
cursor.execute (sql2)
except cx_Oracle.DatabaseError:
print ('Failed \n'+sql2)
#need to do it in chunk as not enough memory and blow up!
while True:
rows = cursor.fetchmany()
if rows == []:
break;
df2=pd.DataFrame.from_records(rows,columns=labels)
df=df.append(df2)
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.