如何知道我的cmd命令是否仍在python中工作

Question

I have collected a large set of text(a online newspaper website) by scraping using Scrapy Framework which I have stored in 'nahidd.txt' file. The txt file size is almost 240MB.

Now In this txt file I have several word redundancy. For example, word 'love' may seen in multiple lines in that txt file. However, I need only one presence of word 'love'

I have used the following code to remove redundancy from my large 'nahidd.txt' file.

file_object = open("nahidd.txt", "r", encoding='utf-8-sig')
file_object_all_text = file_object.read().split()
file_object_redundancy_removed = " ".join(sorted(set(file_object_all_text), key=file_object_all_text.index))
file_object = open("nahidd_pure.txt", "w", encoding='utf-8-sig')
file_object.write(file_object_redundancy_removed)

But the problem is that whenever I put a command in cmd.

scrapy runspider nahidBot.py

It works perfectly fine but It takes forever (since file size is large) and I see a single cursor blinking for hours. It's difficult to understand whether my command is still working or just hanged. I just need to show some kind of text in cmd just like 'line 1 processed', 'line 2 processed' or Percentage of background work done. So that anyone can understand how much work is left or to understand that my command is still working.

Thanks in advance. Nahid

Answer 1

this line performs a sort

file_object_redundancy_removed = " ".join(sorted(set(file_object_all_text), key=file_object_all_text.index))

but uses linear search in the key, which is very bad for performance.

If you don't need to preserve order, just do:

file_object_redundancy_removed = sorted(set(file_object_all_text))

If you need to preserve order "as occurring", which you're trying to emulate with your sort , a faster way would be to store the words you already encountered in an auxiliary set:

marker = set()
file_object_redundancy_removed = []
for w in file_object_all_text:
   if w not in marker:
      marker.add(w)
      file_object_redundancy_removed.append(w)

you now have a list with redundancy removed, and order of first word occurrences preserved.