SQL Python中的存储过程

Question

I am trying to write Python code to imitate the SQL stored produce dynamically. I am just in the begging. I have successfully connected to SQL DB using pyodbc and get data into Pandas df. 1. I have successfully getting all SurveyId from Survey table to construct all SurveyId list in Python. This is the Survey Table: With this code, I get all SurveyId list from Survey table:

def surveyCursor():
    query_GetSurveyId = 'SELECT SurveyId FROM Survey'
    cursor = sql_conn.cursor()
    cursor.execute(query_GetSurveyId)
    row = [item[0] for item in cursor.fetchall()]
    return row

surveyId_list = surveyCursor()
#print(surveyId_list)

2. Now with the next function, I already have this SQL stored procedure as followed:

DECLARE currentQuestionCursor CURSOR FOR
            SELECT *
                    FROM
                    (
                        SELECT
                            SurveyId,
                            QuestionId,
                            1 as InSurvey
                        FROM
                            SurveyStructure
                        WHERE
                            SurveyId = @currentSurveyId
                        UNION
                        SELECT 
                            @currentSurveyId as SurveyId,
                            Q.QuestionId,
                            0 as InSurvey
                        FROM
                            Question as Q
                        WHERE NOT EXISTS
                        (
                            SELECT *
                            FROM SurveyStructure as S
                            WHERE S.SurveyId = @currentSurveyId AND S.QuestionId = Q.QuestionId
                        )
                    ) as t
                    ORDER BY QuestionId;

I need to use this in Python, therefore I created this function:

def CheckInSurvey(SurveyList):
    for surveyId in SurveyList:
        if surveyId < len(SurveyList):
            query_QuestionInSurvey = 'SELECT * FROM (SELECT SurveyId, QuestionId, 1 as InSurvey \
                        FROM SurveyStructure WHERE SurveyId = ' + str(surveyId) + \
                        ' UNION SELECT ' + str(surveyId) + ' as SurveyId, Q.QuestionId, 0 as InSurvey \
                        FROM Question as Q WHERE NOT EXISTS ( SELECT * FROM SurveyStructure as S \
                        WHERE S.SurveyId = ' + str(surveyId) + ' AND S.QuestionId = Q.QuestionId)) as t UNION '
            cursor = sql_conn.cursor()
            cursor.execute(query_QuestionInSurvey)
            row = [item[0] for item in cursor.fetchall()]
            #checkQuestionInSurvey = query_QuestionInSurvey
        else:
            query_QuestionInSurvey_LastRow = 'SELECT * FROM (SELECT SurveyId, QuestionId, 1 as InSurvey \
                        FROM SurveyStructure WHERE SurveyId = ' + str(surveyId) + \
                        ' UNION SELECT ' + str(surveyId) + ' as SurveyId, Q.QuestionId, 0 as InSurvey \
                        FROM Question as Q WHERE NOT EXISTS ( SELECT * FROM SurveyStructure as S \
                        WHERE S.SurveyId = ' + str(surveyId) + ' AND S.QuestionId = Q.QuestionId)) as t ORDER BY QuestionId '
            cursor = sql_conn.cursor()
            cursor.execute(query_QuestionInSurvey_LastRow)
            row = [item[0] for item in cursor.fetchall()]
            #checkQuestionInSurvey = query_QuestionInSurvey_LastRow
        surveyId += 1
    return row

InSurveyList = CheckInSurvey(surveyId_list)

For this function, I want to check the current QuestionId to compared to the Survey Structure table: If current QuestionId is in the Survey Structure table, then set InSurvey to 1 or else set InSurvey to 0.

I want to get a table like this to put in dataframe:

However, when I compile my code, I get this error:

Could you please let me know what can be the issue here? Thank you very much in advance.

Here is the whole Python code:

import pandas as pd
import pyodbc as odbc

# Make a connection to database using pyodbc, then store the query in a variable then put it in pandas dataframe
# To check which ODBC Driver to use:
# 1. From Start menu, go to ODBC Data Sources
# 2. Clieck the Drivers tab, then find SQL Server ODBC Driver in the list of ODBC drivers that installed on the system
sql_conn = odbc.connect('DRIVER={ODBC Driver 17 for SQL Server};SERVER=LAPTOP-NAD8U5G4;DATABASE=Survey_Sample_A19;Trusted_Connection=yes')
query = "SELECT * FROM [Answer]"
sql_to_df = pd.read_sql(query, sql_conn)

#print(sql_to_df.head(3))

# Section 1: Stored Procedue
# Inner loop:
# 1. Iterate in the Survey table to get all SurveyId
# 2. After we get all SurveyId, this will become @currentSurveyId to be used in the loop below in step 3
# Iterate through each Question using "currentQuestionCursor"
# 3. Check if current QuestionId is in SurveyStructure table:
# Yes -> set InSurvey = 1
#     Then, get current QuestionId from Answer table (also SurveyId from Answer table?????), then update Answer string "strQueryTemplateForAnswerColumn", replace with 
#     strColumnsQueryPart AS ANS_Q(currentQuestionID)
# No -> set InSurvey = 0
#     Then, update Answer string "strQueryTemplateForNullColumnn" to
#     NULL AS ANS_Q(currentQuestionID)
# 4. Select SurveyId, AnswerId, then update Answer string
# 5. After the last row, concatenate Answer string "strQueryTemplateForAnswerColumn" with ","
# 6. Remove the cursor "currentQuestionCursor" when finish the Inner

def surveyCursor():
    query_GetSurveyId = 'SELECT SurveyId FROM Survey'
    cursor = sql_conn.cursor()
    cursor.execute(query_GetSurveyId)
    row = [item[0] for item in cursor.fetchall()]
    return row

surveyId_list = surveyCursor()
#print(surveyId_list)

def CheckInSurvey(SurveyList):
    for surveyId in SurveyList:
        if surveyId < len(SurveyList):
            query_QuestionInSurvey = 'SELECT * FROM (SELECT SurveyId, QuestionId, 1 as InSurvey \
                        FROM SurveyStructure WHERE SurveyId = ' + str(surveyId) + \
                        ' UNION SELECT ' + str(surveyId) + ' as SurveyId, Q.QuestionId, 0 as InSurvey \
                        FROM Question as Q WHERE NOT EXISTS ( SELECT * FROM SurveyStructure as S \
                        WHERE S.SurveyId = ' + str(surveyId) + ' AND S.QuestionId = Q.QuestionId)) as t UNION '
            cursor = sql_conn.cursor()
            cursor.execute(query_QuestionInSurvey)
            row = [item[0] for item in cursor.fetchall()]
            #checkQuestionInSurvey = query_QuestionInSurvey
        else:
            query_QuestionInSurvey_LastRow = 'SELECT * FROM (SELECT SurveyId, QuestionId, 1 as InSurvey \
                        FROM SurveyStructure WHERE SurveyId = ' + str(surveyId) + \
                        ' UNION SELECT ' + str(surveyId) + ' as SurveyId, Q.QuestionId, 0 as InSurvey \
                        FROM Question as Q WHERE NOT EXISTS ( SELECT * FROM SurveyStructure as S \
                        WHERE S.SurveyId = ' + str(surveyId) + ' AND S.QuestionId = Q.QuestionId)) as t ORDER BY QuestionId '
            cursor = sql_conn.cursor()
            cursor.execute(query_QuestionInSurvey_LastRow)
            row = [item[0] for item in cursor.fetchall()]
            #checkQuestionInSurvey = query_QuestionInSurvey_LastRow
        surveyId += 1
    return row

InSurveyList = CheckInSurvey(surveyId_list)

And, the SQL stored procedure:

USE [Survey_Sample_A18]
GO
/****** Object:  UserDefinedFunction [dbo].[fn_GetAllSurveyDataSQL]    Script Date: 2/8/2020 4:31:13 PM ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
-- =============================================
-- Author:      <Author,,Name>
-- Create date: <Create Date, ,>
-- Description: <Description, ,>
-- =============================================
CREATE OR ALTER FUNCTION [dbo].[fn_GetAllSurveyDataSQL]() 
RETURNS nvarchar(max)
AS
BEGIN

    DECLARE @strQueryTemplateForAnswerColumn nvarchar(max);
    DECLARE @strQueryTemplateForNullColumnn nvarchar(max);
    DECLARE @strQueryTemplateOuterUnionQuery nvarchar(max);
    DECLARE @currentSurveyId int;

    -- WHEN YOU WRITE DYNAMIC SQL IN STRING VARIABLES
    -- IT'S LIKELY THAT YOU WILL CONCATENATE STRINGS LATER ON IN THE PROCESS
    -- SO, BE CAREFULL OF LEAVING SPACES BEFORE AND AFTER THE QUERY TEXT
    SET @strQueryTemplateForAnswerColumn = '
            COALESCE(
                (
                    SELECT a.Answer_Value
                    FROM Answer as a
                    WHERE
                        a.UserId = u.UserId
                        AND a.SurveyId = <SURVEY_ID>
                        AND a.QuestionId = <QUESTION_ID>
                ), -1) AS ANS_Q<QUESTION_ID> ';

    SET @strQueryTemplateForNullColumnn = ' NULL AS ANS_Q<QUESTION_ID> '

    SET @strQueryTemplateOuterUnionQuery = '
            SELECT
                    UserId
                    , <SURVEY_ID> as SurveyId
                    , <DYNAMIC_QUESTION_ANSWERS>
            FROM
                [User] as u
            WHERE EXISTS
            (
                    SELECT *
                    FROM Answer as a
                    WHERE u.UserId = a.UserId
                    AND a.SurveyId = <SURVEY_ID>
            )
    ';

    DECLARE @strCurrentUnionQueryBlock nvarchar(max);
    SET @strCurrentUnionQueryBlock = ''


    DECLARE @strFinalQuery nvarchar(max);
    SET @strFinalQuery = ''

    --Cursor variables are the only ones to not have an @ in front of their names
    DECLARE surveyCursor CURSOR FOR
                            SELECT SurveyId
                            FROM Survey
                            ORDER BY SurveyId;

    OPEN surveyCursor; -- when opened, the cursor is before the first row
    FETCH NEXT FROM surveyCursor INTO @currentSurveyId; --first row read

    WHILE @@FETCH_STATUS = 0 --AS LONG AS FETCH_STATUS IS EQUAL TO 0 --> there's a row to read
    BEGIN

        -- MAIN LOOP, OVER ALL THE SURVEYS

        -- FOR EACH SURVEY, IN @currentSurveyId, WE NEED TO CONSTRUCT THE ANSWER COLUMN QUERIES

        -- Another iteration, over the questions of the survey
        -- inner loop, another cursor

        --I want a resultset of SurveyId, QuestionId, flag InSurvey indicating whether
        -- the question is in the survey structure
        DECLARE currentQuestionCursor CURSOR FOR
            SELECT *
                    FROM
                    (
                        SELECT
                            SurveyId,
                            QuestionId,
                            1 as InSurvey
                        FROM
                            SurveyStructure
                        WHERE
                            SurveyId = @currentSurveyId
                        UNION
                        SELECT 
                            @currentSurveyId as SurveyId,
                            Q.QuestionId,
                            0 as InSurvey
                        FROM
                            Question as Q
                        WHERE NOT EXISTS
                        (
                            SELECT *
                            FROM SurveyStructure as S
                            WHERE S.SurveyId = @currentSurveyId AND S.QuestionId = Q.QuestionId
                        )
                    ) as t
                    ORDER BY QuestionId;

        DECLARE @currentSurveyIdInQuestion int;
        DECLARE @currentQuestionID int;
        DECLARE @currentInSurvey int;

        OPEN currentQuestionCursor;
        --When fetching into local variable, the column order of the select clause must be followed
        FETCH NEXT FROM currentQuestionCursor INTO @currentSurveyIdInQuestion,
                                                    @currentQuestionID,
                                                    @currentInSurvey;

        DECLARE @strColumnsQueryPart nvarchar(max);

        SET @strColumnsQueryPart = '';
        WHILE @@FETCH_STATUS = 0
        BEGIN
            --the "global" variable @@FETCH_STATUS GETS LOCALISED BETWEEN BEGIN --- END BLOCK

            --INNER LOOP IETERATES OVER THE QUESTION

            --IS THE CURRENT QUESTION (inner loop) IN THE CURRENT SURVEY (outer loop)

            IF @currentInSurvey = 0 -- CURRENT QUESTION IS NOT IN THE CURRENT SURVEY
            BEGIN
                --THEN BLOCK
                -- SPECIFICATION: THE VALUES IN THIS COLUMN WILL BE NULL
                SET @strColumnsQueryPart = @strColumnsQueryPart +
                        REPLACE(@strQueryTemplateForNullColumnn, '<QUESTION_ID>',
                                @currentQuestionID);
            END
            ELSE
            BEGIN
                SET @strColumnsQueryPart = @strColumnsQueryPart +
                    REPLACE(@strQueryTemplateForAnswerColumn, '<QUESTION_ID>',
                            @currentQuestionID);
            END;

            FETCH NEXT FROM currentQuestionCursor INTO @currentSurveyIdInQuestion,
                                                    @currentQuestionID,
                                                    @currentInSurvey;

        IF @@FETCH_STATUS = 0
        BEGIN
            -- Place a comma between column statements, except for the last one
            SET @strColumnsQueryPart = @strColumnsQueryPart + ' , ';
        END;

        END; -- END OF CLOSE INNER LOOP WHILE
        CLOSE currentQuestionCursor;
        DEALLOCATE currentQuestionCursor;


        --BACK IN THE OUTER LOOP OVER SURVEYS
        SET @strCurrentUnionQueryBlock = 
            REPLACE(@strQueryTemplateOuterUnionQuery,
                    '<DYNAMIC_QUESTION_ANSWERS>',
                    @strColumnsQueryPart);

        SET @strCurrentUnionQueryBlock = 
            REPLACE(@strCurrentUnionQueryBlock, 
                        '<SURVEY_ID>', @currentSurveyId);

        SET @strFinalQuery = @strFinalQuery + @strCurrentUnionQueryBlock

        FETCH NEXT FROM surveyCursor INTO @currentSurveyId;

        IF @@FETCH_STATUS = 0 
        BEGIN
            SET @strFinalQuery = @strFinalQuery + ' UNION ' ;
        END;

    END;

    CLOSE surveyCursor;
    DEALLOCATE surveyCursor;

    --calling the system store procedure sp_executesql
    --recommended by Microsoft should the text of your dunamic query > 4000 chars
     RETURN @strFinalQuery;


END

Answer 1

In CheckInSurvey the last UNION in query_QuestionInSurvey might cause a problem.

def CheckInSurvey(SurveyList):
for surveyId in SurveyList:
    if surveyId < len(SurveyList):
        query_QuestionInSurvey = 'SELECT * FROM (SELECT SurveyId, QuestionId, 1 as InSurvey \
                    FROM SurveyStructure WHERE SurveyId = ' + str(surveyId) + \
                    ' UNION SELECT ' + str(surveyId) + ' as SurveyId, Q.QuestionId, 0 as InSurvey \
                    FROM Question as Q WHERE NOT EXISTS ( SELECT * FROM SurveyStructure as S \
                    WHERE S.SurveyId = ' + str(surveyId) + ' AND S.QuestionId = Q.QuestionId)) as t UNION '
#       See HERE! ----------------------------------------------------------------------------------^
        cursor = sql_conn.cursor()
        cursor.execute(query_QuestionInSurvey)
...

Answer 2

I used another way to execute my SQL query: query_questionInSurvey = pd.read_sql_query(InSurveyList, sql_conn) Instead of cursor.execute()