I have a CSV file with about 700 or so rows and 8 columns, the last column however, has a very big block of text (with enough for multiple long paragraphs inside each).
I'd like to implement through python a text-search function that gives me back all the lines that have text that matches from inside the data from the 8th column (meaning it'd need to go through the whole thing).
What could possibly be the quickest way to approach this and minimise search-time?
You could dump your csv file into an sqlite database and use sqlite's full text search capabilities to do the search for you.
This example code shows how it could be done. There are a few things to be aware of:
import csv
import sqlite3
import sys
def create_table(conn, headers, name='mytable'):
cols = ', '.join([x.strip() for x in headers])
stmt = f"""CREATE VIRTUAL TABLE {name} USING fts5({cols})"""
with conn:
conn.execute(stmt)
return
def populate_table(conn, reader, ncols, name='mytable'):
placeholders = ', '.join(['?'] * ncols)
stmt = f"""INSERT INTO {name}
VALUES ({placeholders})
"""
with conn:
conn.executemany(stmt, reader)
return
def search(conn, term, headers, name='mytable'):
cols = ', '.join([x.strip() for x in headers])
stmt = f"""SELECT {cols}
FROM {name}
WHERE {name} MATCH ?
"""
with conn:
cursor = conn.cursor()
cursor.execute(stmt, (term,))
result = cursor.fetchall()
return result
def main(path, term):
result = 'NO RESULT SET'
try:
# Create an in-memory database.
conn = sqlite3.connect(':memory:')
with open(path, 'r') as f:
reader = csv.reader(f)
# Assume headers are in the first row
headers = next(reader)
create_table(conn, headers)
ncols = len(headers)
populate_table(conn, reader, ncols)
result = search(conn, term, headers)
finally:
conn.close()
return result
if __name__ == '__main__':
print(main(*sys.argv[1:]))
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.