簡體   English   中英

程序類被卡住/空閑,並且在Anaconda /命令行提示符中進行第一次調用后不執行其余調用,但在Spyder中有效

[英]Program class is stuck/idle and does not execute remaining calls after 1st call in Anaconda/Command Line Prompt but works in Spyder

我正在嘗試使用anaconda提示符運行我的python腳本。 它在第一次通話時運行平穩,但在此停止。 我在Spyder上嘗試過,它可以工作,但我希望它在anaconda提示符或命令行下工作。 有什么原因嗎?

from decompress import decompress
from reddit import reddit
from clean import clean
from wikipedia import wikipedia

def main():
    dir_of_file = r"D:\Users\Jonathan\Desktop\Reddit Data\Demo\\"
    print('0. Path: ' + dir_of_file)
    reddit_repo = reddit()
    wikipedia_repo = wikipedia()
    pattern_filter = "*2007*&*2008*"
    print('1. Creating data lake')
    reddit_repo.download_files(pattern_filter,"https://files.pushshift.io/reddit/submissions/",dir_of_file,'s') 
    reddit_repo.download_files(pattern_filter,"https://files.pushshift.io/reddit/comments/",dir_of_file,'c')         

if __name__ == "__main__":
    main()

RS Downloaded是正在運行的以下代碼行:

reddit_repo.download_files(pattern_filter,"https://files.pushshift.io/reddit/submissions/",dir_of_file,'s') 

在此處輸入圖片說明 更新:

添加了類/功能

class reddit:

    def multithread_download_files_func(self,list_of_file):
        filename = list_of_file[list_of_file.rfind("/")+1:]
        path_to_save_filename = self.ptsf_download_files + filename
        if not os.path.exists(path_to_save_filename): 
            data_content = None
            try:
                request = urllib.request.Request(list_of_file)
                response = urllib.request.urlopen(request)
                data_content = response.read()
            except urllib.error.HTTPError:
                print('HTTP Error')
            except Exception as e:
                print(e)
            if data_content:
                with open(path_to_save_filename, 'wb') as wf:    
                    wf.write(data_content)                 
                    print(self.present_download_files + filename)                        

    def download_files(self,filter_files_df,url_to_download_df,path_to_save_file_df,prefix):
        #do some processing
        matching_fnmatch_list.sort()

        p = ThreadPool(200)
        p.map(self.multithread_download_files_func, matching_fnmatch_list)

下載需要花費大量時間。 我更改了網絡,它按預期工作。 因此,cmd或anaconda提示符沒有問題

暫無
暫無

聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.

 
粵ICP備18138465號  © 2020-2024 STACKOOM.COM