![](/img/trans.png)
[英]How do I download and upload files as effectively as possible using Python?
[英]How do I upload many files at the time to cloud files with Python?
我正在使用cloudfile模块将文件上传到rackspace云文件,使用类似这样的伪代码:
import cloudfiles
username = '---'
api_key = '---'
conn = cloudfiles.get_connection(username, api_key)
testcontainer = conn.create_container('test')
for f in get_filenames():
obj = testcontainer.create_object(f)
obj.load_from_filename(f)
我的问题是我要上传很多小文件,这种方式需要太长时间。
埋藏在文档中,我看到有一个类ConnectionPool ,据说可以用来上传并行文件。
有人可以说明我如何让这段代码一次上传多个文件?
ConnectionPool
类适用于多线程应用程序,该应用程序偶尔必须向rackspace发送内容。
这样您就可以重用连接,但如果您有100个线程,则不必保持100个连接打开。
您只是在寻找多线程/多处理上传器。 以下是使用multiprocessing
库的示例:
import cloudfiles
import multiprocessing
USERNAME = '---'
API_KEY = '---'
def get_container():
conn = cloudfiles.get_connection(USERNAME, API_KEY)
testcontainer = conn.create_container('test')
return testcontainer
def uploader(filenames):
'''Worker process to upload the given files'''
container = get_container()
# Keep going till you reach STOP
for filename in iter(filenames.get, 'STOP'):
# Create the object and upload
obj = container.create_object(filename)
obj.load_from_filename(filename)
def main():
NUMBER_OF_PROCESSES = 16
# Add your filenames to this queue
filenames = multiprocessing.Queue()
# Start worker processes
for i in range(NUMBER_OF_PROCESSES):
multiprocessing.Process(target=uploader, args=(filenames,)).start()
# You can keep adding tasks until you add STOP
filenames.put('some filename')
# Stop all child processes
for i in range(NUMBER_OF_PROCESSES):
filenames.put('STOP')
if __name__ == '__main__':
multiprocessing.freeze_support()
main()
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.