Python/SQL - WHERE clause not applying to all Rows

Question

I am having a problem with the following code in a Python program I built. The program pulls data from a database and puts it into an Excel spreadsheet based on a list of reference numbers 'GrantRefNumber' that they input.

It works, but for some reason only the FIRST reference number has the 'a.reporting_period_id like 'none-' applied to it, the rest do not.

I am using a variable substitution in the SQL code to put the list of reference numbers into the string.

Any help would be much appreciated!

SQL:

"SELECT a.fa_reference as [GrantRefNumber], 
a.fa_name as [Award Title], 
a.location as [RO], 
ISNULL(cat.grant_department_name, '') as [Department], 
a.funding_start_date as [Start Date], 
a.funding_end_date as [End Date], 
a.[pi]    as [PI ID],
a.pi_initials  as [PI Name], 
a.pi_surname as [PI Surname],  
r1_2 as [Type], 

DATEADD(s, cast(last_submitted_date as int), '1970-01-01 00:00:00') as [Submitted Date]  

from keywordagreements a inner join entries_publications p on a.id =  
p.agreement_id left outer join mrc_categories cat  on a.origid = cat.id and cat.centre not in ('2') 

where a.[pi] NOT LIKE 'S%' and response_code not like 'Test' and Closed is null  
and a.reporting_period_id like 'none-'
and a.funding_organisation not like '%UKSA%' and {}' 

Order by [RO], [PI ID], [GrantRefNumber]".format(finalList)

finalList (variable substitution):

The finalList is a list of Reference Numbers that I get from the user in Python called 'items'

items = dfCall['GrantRefNumber'].values.tolist()
refList = " OR ".join(["a.fa_reference LIKE '%s'" % num for num in items])
finalList = refList[:-1]

I am using PYODBC to extract the data.

The SQL in my code looks like this (I removed the quote marks (and some of the columns) in the previous code to make it easier to read):

stringQ = "SELECT a.fa_reference as [GrantRefNumber], a.fa_name as [Award Title]," \
          " a.location  as [RO], ISNULL(cat.grant_department_name, '') as [Department]," \
          " a.funding_start_date as [Start Date], a.funding_end_date        as [End Date]," \
          " a.[pi]  as [PI ID], a.pi_initials   as [PI Name]," \
          " a.pi_surname as [PI Surname], " \
          " r1_2 as [Type], ISNULL(r1_2_1, '')  as [PubMed ID]," \
          " r1_2_2  as [Author], r1_2_3 as [Publication], ISNULL(r1_2_4, '') as [Journal]," \
          " ISNULL(r1_2_8, '') as [Month], ISNULL(r1_2_9, '') as [Year], ISNULL(r1_2_4_1, '') as [Conference]," \
          " ISNULL(r1_2_36, '') as [PubMed Central ID], ISNULL (r1_2_19, '') as [DOI]," \
          " case when nullif(r1_2_1,'') is not null then 'http://europepmc.org/abstract/MED/' + r1_2_1 else case when" \
          " nullif(r1_4,'') is not null then r1_4 else case when nullif(r1_2_19,'') is not null then" \
          " 'http://dx.doi.org/' + r1_2_19 else isnull(r1_2_1,'') end  end end as [URL], ISNULL(r1_2_21, '') " \
          "as [ISBN]," \
          " ISNULL(r1_2_30, '') as [ISBN (Electronic)], " \
          " ISNULL(r1_2_25, '') as [Chapter Number], " \
          "ISNULL(r1_2_26, '')" \
          " as [Chapter Title], ISNULL(r1_2_27, '') as [Chapter Author]," \
          " ISNULL(r1_2_29, '') as [ISSN (Print)], ISNULL(r1_2_32, '')  as [ISSN (Digital)], " \
          "ISNULL(r1_2_31, '')  as [Web of Science ID], ISNULL(r1_2_34, '') as [Scopus ID], " \
          "ISNULL(r1_2_35, '')  as [arXiv DepositID]," \
          " ISNULL(r1_2_38, '') as [Bibcode], ISNULL(r1_2_39, '') as [Ethos], ISNULL(r1_2_43, '')   as [NASA-ADS ID]," \
          " ISNULL(r1_2_46, '') as [Inspire], ISNULL(r1_2_40, '') as [PMC Manuscript ID], ISNULL(r1_2_45, '')" \
          " as [ORCID Work Putcode]," \
          " ISNULL(r1_2_61, '') as [OpenAire Access License], ISNULL(r1_2_52, '') " \
          "as [In EPMC?], ISNULL(r1_2_53, '') as [In PMC?]," \
          " ISNULL(r1_2_51, '') as [EPMC Open Access], " \
          " DATEADD(s, cast(last_submitted_date as int), '1970-01-01 00:00:00') as [Submitted Date] " \
          "from keywordagreements a inner join entries_publications p on a.id = " \
          "p.agreement_id left outer join mrc_categories cat " \
          "on a.origid = cat.id and cat.centre not in ('2') where a.[pi] NOT LIKE 'S%' and " \
          "response_code not like 'Test' and Closed is null " \
          "and a.reporting_period_id like 'none-' " \
          "and a.funding_organisation not like '%UKSA%' and {}' " \
          "Order by [RO], [PI ID], [GrantRefNumber]".format(finalList)

Answer 1

I'm not seeing where your format command substitutes. Here are a few examples of the format command:

'This is a {} string.'.format('formatted')
'This is a {0} {1}.'.format('formatted', 'string')
'This is a {replace_me} {replace_me_2}.'.format(replace_me='formatted', replace_me_2='string')

Outputs This is a formatted string.

You'll need to modify you SQL to insert finalList into the SQL. Since we can't see your whole code, it also appear like you might want to use triple-quote formatting for multi-line support:

sql = """
    SELECT * FROM table
    WHERE blah = '{string_to_compare}'
""".format(string_to_compare='blah')

Be very careful, you're open to SQL injection when using string substitution in any SQL query. Good luck!

Answer 2

Ok, I think I have fixed it. I put the {}' in brackets ie ({}') as it was treating the other reference numbers as a separate statement because of the OR command.