avoid data loss when stopping python script that runs in windows command prompt

Question

I've written a Python-script, that scrapes certain websites. I run it from windows command line and use the command line to direct the "print()"-output of the script into a txt-file.

Like this:

MyScraper.py > outputOfMyScraper.txt.

Now and then I have to kill/stop the script before the run is complete (I close the windows command line). 9 of 10 times this has no impact on the outputOfMyScraper.txt-file, but now and then the content of the file will be deleted when closing the command line.

Is there - besides changing the code in the scraper to write content directly to a csv-file and avoid the windows command line totally - a way of stopping the script without the risk of losing data?

Thank you very much.

Answer 1

You can force Python to switch standard output to unbuffered mode by adding -u switch:

python -u MyScraper.py > outputOfMyScraper.txt

Another way is to set the PYTHONUNBUFFERED environment variable to non-empty string:

set PYTHONUNBUFFERED=1
MyScraper.py > outputOfMyScraper.txt