psycopg2：如何将 Python 应用程序中的数据添加到 Postgresql

Question

This is a part of a project that captures news headlines, tracks leanings, and returns sentiment analysis (ie. positive, negative, and neutral). Everything works on the following code except for simply adding the captured data to a PostgreSQL table (FYI: CNN and USAToday are in the works as well, as I respect all news sources).

Currently, the following error occurs:

"Exception has occurred: SyntaxError syntax error at or near "%" "

I have tried to find examples of my specific situation, but all examples I see have the actual results in the insertion value, instead of the python values that contain the needed data. I have attached the original table creation code as well:

--create table
create table news_data (
timestamp timestamp,
user varchar(75),
url text,
site varchar(75),
lean varchar(75),
source varchar(75),
headline varchar(1000),
results text,
positive float,
negative float,
neutral float,

)

PYTHON CODE:

#import the needed libraries
import os
import nltk
#nltk.download() #Only use once
import datetime
import getpass
import requests
import time
import numpy as np
import pandas as pd
import pandasql as psql
import lxml
from bs4 import BeautifulSoup
from nltk.sentiment.vader import SentimentIntensityAnalyzer as SIA
#from nltk.corpus.reader.plaintext import PlaintextCorpusReader as PCR
import psycopg2 as p2

#Application process
ts = datetime.datetime.now().date() #needs to be converted to a string
timestamp = ts.strftime("%d-%b-%Y (%H:%M:%S.%f)")
user = getpass.getuser()
url = 'https://www.foxnews.com/'
site = 'Fox News'
lean = 'Conservative'
source = requests.get(url)
soup = BeautifulSoup(source.content, 'lxml') #'source.content' is critical to success here
headline = soup.find('h2', class_='title title-color-default').text #apply sntiment analysis to headline
vader = SIA()
sentiment_dict = vader.polarity_scores(headline)
results = vader.polarity_scores(headline)
positive = (sentiment_dict['pos']*100)
negative = (sentiment_dict['neg']*100)
neutral = (sentiment_dict['neu']*100)


#lists
#sql_list = (timestamp, user, url, site, lean, headline, results, positive, negative, neutral)

#Postgresql connection
db_connection = p2.connect(user = 'PLACEHOLDER', password='PLACEHOLDER', database='PLACEHOLDER')
cursor = db_connection.cursor()
insertion = cursor.execute('''
INSERT INTO news_data VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)
''')

record_to_insert = (timestamp, user, url, site, lean, headline, source, results, positive, negative, neutral)
cursor.execute(insertion, record_to_insert)


cursor.close()
db_connection.close()

        
#Testing
#print(headline)
#print(negative)

Even after changing the final lines into the following, I still get the following error:

Exception has occurred: SyntaxError syntax error at or near "%" LINE 1: ...ce, results, positive, negative, neutral) VALUES (%s, %s, %s...

Changed Code:

cursor.execute("INSERT INTO news_data (timestamp, url, site, lean, headline, source, results, positive, negative, neutral) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s), (timestamp, url, site, lean, headline, results, positive, negative, neutral)")

cursor.commit()

cursor.close()

db_connection.close()
        
#Testing
#print(headline)
#print(negative)

Answer 1

Without looking at each data type in your statement, I can see that a few of them are strings. Make sure you're surrounding string-type values with single-quotes where necessary.

insertion = cursor.execute('''
INSERT INTO news_data VALUES ('%s','%s', ...)
''')

Answer 2

ah, here's the relevant part of the code

#Postgresql connection
db_connection = p2.connect(user = 'PLACEHOLDER', password='PLACEHOLDER', database='PLACEHOLDER')
cursor = db_connection.cursor()

# hmm, this won't work.  You need to provide values for all those placeholders
insertion = cursor.execute('''
INSERT INTO news_data VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)
''')

# oh, i see what you're up to.  You want this:

insertion = 'INSERT INTO news_data....'

record_to_insert = (timestamp, user, url, site, lean, headline, source, results, positive, negative, neutral)

cursor.execute(insertion, record_to_insert)

cursor.close()

# most likely, you'll also need
db_connection.commit()

db_connection.close()

Answer 3

Basically, psycopg2 does not support adding stored values from Python variables. You have to save the data to a CSV file first, have the code take the data from the CSV file, upload the data to PostgreSQL, then finally delete the CSV file once the process is complete.

Hopefully, this will help someone in the future who is trying to accomplish something similar.