Python and sqlite3 throwing an error: sqlite3.OperationalError: near “s”: syntax error

Question

I'm trying to use Python and BeautifulSoup to scrape some web info, iterate through it and then insert some pieces into a sqlite3 DB. But I keep coming up with this error:

File "/Users/Chris/Desktop/BS4/TBTfile.py", line 103, in TBTscrape c.execute(item) sqlite3.OperationalError: near "s": syntax error

This same code works fine on a very similar scraper and does not throw these errors. Here's the portion of it:

listicle.append("INSERT INTO headlines (heds, url, image_loc, time, source) VALUES ('" + TBTheadline + "', '" + TBTurl + "', '" + imageName + "', '" + TBTpostDate + "', '" + source + "')")

    else:
        print "TBT item already in database"

print listicle

for item in listicle:
    c.execute(item)
    conn.commit()
    row = c.fetchall()
    print "This has been inserted succcessfully: ", item

Answer 1

You are concatenating the collected data into your SQL statements. Never do that, it is the mother of all anti-patterns . Aside from the problems you are seeing (probably due to a ' or similar character in the scraped HTML), you have a gaping security hole in your code (may or may not matter in your case).

Anyway, sqlite3 has a nice way of doing exactly what you want: executemany . In your case

listicle.append((TBTheadline,TBTurl,imageName,TBTpostDate,source))

conn.executemany("INSERT INTO headlines (heds, url, image_loc, time, source) VALUES (?,?,?,?,?)", listicle)