![](/img/trans.png)
[英]how to merge same valued data frames with different data types in python
[英]Python - loop through same query with different variables, merge data frames
我在 SAS 中有一個查詢,其中我使用宏變量使用不同的變量重復對 Teradata 的查詢。 我們有 5 個數據庫,每個狀態一個,我在其中運行相同的查詢,但使用變量更新狀態,然后修復所有數據集。 我正在尋求有關如何在 python 中執行此操作的幫助。
循環遍歷 {state1, state2, state3, state4, state5} 並將每個查詢保存為 {stateX}_df 然后合並所有
import teradata as td
import pandas as pd
from teradata import tdodbc
udaExec = td.UdaExec(appConfigFile="udaexec.ini")
with udaExec.connect("${dataSourceName}", LoginTimeout=120) as session:
query1 = """database my_db_{state1};"""
query2 = """
select distinct
{state1}, item_a, item_b
from table
"""
session.execute(query1)
session.execute(query2)
{stateX}_df = pd.read_sql(query2), session)
不確定你使用的是 python 2 還是 python 3。如果你可以使用 python 3.6 或更高版本,也許像下面這樣的東西可以工作?
import teradata as td
import pandas as pd
udaExec = td.UdaExec(appName="test", version="1.0", logConsole=False)
with udaExec.connect(
method="odbc",
system="host",
username="username",
password="password",
driver="drivername"
) as conn:
state_dataframes = []
STATES = ["state1", "state2", "state3", "state4", "state5"]
for state in STATES:
sql = f"select distinct {state}, item_a, item_b from my_db_{state}.table;"
state_dataframes.append(pd.read_sql(sql, conn))
combined_data = pd.concat(state_dataframes)
這沒有經過測試,但希望它能讓你朝着正確的方向前進。
我能夠在單個測試查詢上完成這項工作,這真的很有幫助,所以謝謝@andrew madsen
我尚未解決的是如何在我使用的多個查詢中執行此操作。 我一直在閱讀有關游標和連接的內容,我認為這會讓我到達那里。
import teradata as td
import pandas as pd
from teradata import tdodbc
udaExec = td.UdaExec(appConfigFile="udaexec.ini")
with udaExec.connect("${dataSourceName}") as session:
state_dataframes = []
STATES = ["IL", "TX", "MT", "OK", "NM"]
for state in STATES:
sql = """
select top 10
'{}' as state
,a.*
from my_db_{}.table a
""".format(state,state)
state_dataframes.append(pd.read_sql(sql, session))
all_states_df = pd.concat(state_dataframes)
這是使用易失性表的改進版本: Python SQL loop variables through multiple queries
udaExec = td.UdaExec(appConfigFile="udaexec.ini")
with udaExec.connect("${dataSourceName}") as session:
state_dataframes = []
STATES = ["state1", "state2", "state3", "state4", "state5"]
for state in STATES:
query1 = """database my_db_{};"""
query2 = """
create set volatile table v_table
,no fallback, no before journal, no after journal as
(
select top 10
'{}' as state
,t.*
from table t
)
with data
primary index (dw_key)
on commit preserve rows;
"""
query3 = """
create set volatile table v_table_2
,no fallback, no before journal, no after journal as
(
select t.*
from v_table t
)
with data
primary index (dw_clm_key)
on commit preserve rows;
"""
query4 = """
select t.*
from v_table_2 t
"""
session.execute(query1.format(state))
session.execute(query2.format(state))
session.execute(query3)
session.execute(query4)
state_dataframes.append(pd.read_sql(query4, session))
session.execute("DROP TABLE v_table")
session.execute("DROP TABLE v_table_2")
all_states_df = pd.concat(state_dataframes)
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.