multiple python scripts one text file

Question

I want to know if it can be done.

I have a script made in python that takes a random line from a text file, process it and returns a modified line, i insert it in the file and that's it.

My problem is that i want to expand the script, make it multithreading, is there a chance that the script wont be accurate in writing the line to the text file, i mean the script runs fast and i don't know if the file system will crash or not, has anyone used text files for this kind of thing, what would you recommend instead of text files?

If i move all the data to a database mysql/sql(whatever) will i loose from the speed much? I mean reading a text file is done quite quick, if i connect to the database and get the data after that process it, it takes time i presume, or wont it if i install a mysql server on my pc..?

Please keep in mind that the file is growing extremely fast extremely big.

Answer 1

How big of a file are you talking about? You said it's growing quickly. The nice thing about putting something in a db like postgres is write-ahead logging. There are several nice python libraries for interacting with db's like psycopg2 for postgres. Do you have a need to query information in the future, in which case a db may be worth any minor loss in speed.

http://en.wikipedia.org/wiki/Write-ahead_logging

http://initd.org/psycopg/