我应该怎么做才能使程序运行更快？

Question

I have a python program which processes a html page and creates a dictionary of urls as key and md5sum of the file as a value. The dictionary length is 6000. Each url is a zip file which is downloaded in to machine, each time checking md5sum after the file is downloaded. The total size of all the files that are to be downloaded is 572 GB.

The URLs is a dictionary which has download links as key and md5sum of file as value

The code is

        DownloadAllURLs(URLs)

        def DownloadAllURLs(URLs):
            for eachurl in URLs:
                if os.path.isfile(eachurl):
                     print eachurl, "already exists"
                else:
                     print "Going to Download",eachurl
                     Download(eachurl)
                     CheckMd5(eachurl,URLs(eachurl))

        def Download(eachurl):
            command='sudo wget --user=abc --password=xyz'
            command=command+" "+url
            print command
            result=subprocess.Popen(command,shell=True,stdout=subprocess.PIPE,
            stderr=subprocess.PIPE)
            out, err=result.communicate()

        def CheckMd5(url,tail,md5sum):
            command=['md5sum',tail]
            result=subprocess.Popen(command,stdout=subprocess.PIPE,stdin=subprocess.PIPE)
            md5, err=result.communicate()
            if(md5[:32]==md5sum):
                print "The",tail,"is downloaded successufully with correct md5"
            else:
                print "The",tail,"is not downloaded correcty wrong md5"
               WriteWarcFile(url,tail)
               CheckMd5(url,tail,md5sum)

The above code downloads all the 6000 zip files for me, but the server from where i am downloading is very slow and i get only 40-60 kbps some times when downloading ..

I am using the above code to download like 1-3 terrabytes of data.... I want to parallelize my code in python (so the time taken to process will be reduced) but i am not sure whether to use multithreading or multiprocessing or anything else.

I am reading the tutorials but not sure how to proceed. Thank you

Edited:

Thanks for all the replies, the main question i want to ask is how to apply multithreading/multithreading in cases like this. Suppose i am doing some operations on every URL rather than downloading it like the below code, can i make it any faster using multithreading or mutlprocessing

    from urlparse import urlparse
    ProcessAllURLs(URLs)
    def ProcessAllURLs(URLs):
        for eachurl in URLs:
                x=urlparse(eachurl)
                print eachurl.netloc

Answer 1

由于处理是受IO限制的，因此应该可以使用python多线程处理-全局解释器锁定不会对性能产生太大影响