如何将来自 SQL 服务器的 Python 结果保存到 SQL 表中

Question

Please help me to save the results of the Python query through the SQL server - to the SQL table.

This script works perfect if the Python script always gives the same number of columns:

insert INTO [dbo].[tmp]
           ([col1]
           ,[col2])
exec sp_execute_external_script  
       @language =N'Python'    
       ,@script=N'
import pandas as pd
df = pd.DataFrame({
    "name": ["John Smith", "Jane Doe", "Joe Schmo"],
    "address": ["123 Main St.", "456 Maple Ave.", "789 Broadway"]})
'
,@output_data_1_name =  N'df'

The problem is, the Python script can produce a different number of columns each time, including columns with new names.

Script:

select * 
into tmp
from
(exec sp_execute_external_script  
       @language =N'Python'    
       ,@script=N'
import pandas as pd
df = pd.DataFrame({
    "name": ["John Smith", "Jane Doe", "Joe Schmo"],
    "address": ["123 Main St.", "456 Maple Ave.", "789 Broadway"],
    "age": [34, 28, 51]
})
'
,@output_data_1_name =  N'df')

Gives an error message:

Msg 156, Level 15, State 1, Line 4
Incorrect syntax near the keyword 'exec'.
Msg 102, Level 15, State 1, Line 14
Incorrect syntax near ')'.

and does not fit to my needs, because I need to add the results to an existing table, and not to recreate each time.

I tried the following: 1. run the Python script for the 1st time and pull out all the column names, which will then be inserted - into @columns variable 2. if existing table (tmp) doesn't have some columns from the step 1 - update it 3. run the same Python script for the 2nd time, but instead of

insert INTO [dbo].[tmp]
           ([col1]
           ,[col2])

use:

insert INTO [dbo].[tmp]
           (@columns)
exec sp_execute_external_script  
       @language =N'Python'    
       ,@script=N'
import pandas as pd
df = pd.DataFrame({
    "name": ["John Smith", "Jane Doe", "Joe Schmo"],
    "address": ["123 Main St.", "456 Maple Ave.", "789 Broadway"],
    "age": [34, 28, 51]
})
'
,@output_data_1_name =  N'df'

Where @columns is a list of columns obtained in step 1. an error message occurs:

Msg 207, Level 16, State 1, Line 6
Invalid column name '@columns'.

Then I try to do it through exec (). To avoid quotation marks problems, I create separate variables for each query value.

declare @for_language nvarchar(255) = 'Python'
declare @for_output_data_1_name nvarchar(255) = 'df'
declare @for_script nvarchar(max) = 'import pandas as pd
df = pd.DataFrame({
    "name": ["John Smith", "Jane Doe", "Joe Schmo"],
    "address": ["123 Main St.", "456 Maple Ave.", "789 Broadway"],
    "age": [34, 28, 51]
})'

exec('insert INTO [dbo].[tmp] (' + @columns + ') exec sp_execute_external_script  
       @language =' + @for_language + ',@script=' + @for_script + ',@output_data_1_name ' + @for_output_data_1_name)

an error message occurs:

Msg 102, Level 15, State 1, Line 4
Incorrect syntax near 'pandas'.

If I run the same query through select instead of exec, it forms the correct query string:

insert INTO [dbo].[tmp] ([col1]
           ,[col2]
           ,[col3]) exec sp_execute_external_script   
       @language =Python,@script=import pandas as pd 
df = pd.DataFrame({ 
    "name": ["John Smith", "Jane Doe", "Joe Schmo"], 
    "address": ["123 Main St.", "456 Maple Ave.", "789 Broadway"], 
    "age": [34, 28, 51] 
}),@output_data_1_name df

This is the third time I create the topic. I'm looking for the solution for many days but seriously stucked. If some more information is needed, please let me know.

Thank you so much to everyone, who will share any ideas, thoughts on how to get the results. Even if there is no ready-made solution.

Answer 1

The answer is - escape single quotes in the Python construction script. 1) I should have written the script the following way:

declare @columns nvarchar(255) = '[col1]
           ,[col2]
           ,[col3]'
declare @for_language nvarchar(255) = 'N''Python'''
declare @for_output_data_1_name nvarchar(255) = 'N''df'''
declare @for_script nvarchar(max) = 'N''
import pandas as pd
df = pd.DataFrame({
    "name": ["John Smith", "Jane Doe", "Joe Schmo"],
    "address": ["123 Main St.", "456 Maple Ave.", "789 Broadway"],
    "age": [34, 28, 51]
})'''

exec ('insert INTO [dbo].[tmp] (' + @columns + ') exec sp_execute_external_script  
       @language =' + @for_language + ',@script=' + @for_script + ',@output_data_1_name =' + @for_output_data_1_name)

2) temporarily substitute exec by select 3) extract the constructed script from the string 4) run the script from the string separately from the constructor - whether it works. 5) if doesn't - to repare the script: primarily on the string level and then in the constructor