[英]Python Threading - How do i use it to run multiple tasks at time?
Im new to Python, and im having a ruge help from stackoverflow comunity in order to migrate my shellscript to python. 我是Python的新手,并且我从stackoverflow社区获得了巨大的帮助,以便将我的shellscript迁移到python。 But again im struggling on how i can implement threading since this script runs over ax results, would be faster to put it to run with, for example, the scripts return 120 servers to run, i would like to run 5 at time and have a queue. 但是再次由于该脚本在ax结果上运行而使我难以实现线程,因此使其运行起来更快,例如,脚本返回120台服务器来运行,我想一次运行5台服务器并拥有一个队列。
The method that i want t run on thread is after the condition bellow: ( i marked with comments ) 我不希望在线程上运行的方法是在以下条件之后:(我用注释标记)
if checkServer.checkit(host,port):
Bellow, is the extract_adapter.py file content: 吼叫,是extract_adapter.py文件的内容:
import psycopg2
import urllib2
import base64
import sys
import re
import lxml.html as LH
import checkServer
def extractAdapter(env,family,iserver,login,password,prefix,proxyUser,proxyPass,proxyHost,service):
print "Starting on \t"+iserver
proxy_auth = "http://"+proxyUser+":"+proxyPass+"@"+proxyHost
proxy_handler = urllib2.ProxyHandler({"http": proxy_auth})
opener = urllib2.build_opener(proxy_handler)
urllib2.install_opener(opener)
request = urllib2.Request("http://"+iserver+"/invoke/listRegisteredAdapters")
base64string = base64.encodestring('%s:%s' % (login, password)).replace('\n', '')
request.add_header("Authorization", "Basic %s" % base64string)
response = urllib2.urlopen(request)
html = response.read()
doc = LH.fromstring(html)
tds = (td.text_content() for td in doc.xpath("//td[not(*)]"))
for adapterType, adapterDescription in zip(*[tds]*2):
proxy_auth = "http://"+proxyUser+":"+proxyPass+"@"+proxyHost
proxy_handler = urllib2.ProxyHandler({"http": proxy_auth})
opener = urllib2.build_opener(proxy_handler)
opener = urllib2.build_opener()
urllib2.install_opener(opener)
request = urllib2.Request("http://"+iserver+service+""+adapterType)
base64string = base64.encodestring('%s:%s' % (login, password)).replace('\n', '')
request.add_header("Authorization", "Basic %s" % base64string)
response = urllib2.urlopen(request)
html2 = response.read()
doc = LH.fromstring(html2)
tds = (td.text_content() for td in doc.xpath("//td[not(*)]"))
for connectionAlias,packageName,connectionFactoryType,mcfDisplayName,connectionState,hasError in zip(*[tds]*6):
cur.execute("INSERT INTO wip.info_adapter (env,family,iserver,prefix,package,adapter_type,connection_name,status) values (%s,%s,%s,%s,%s,%s,%s,%s)",
(env,family,iserver,prefix,packageName,adapterType,connectionAlias,connectionState))
con.commit()
################################################################################
def extract(env):
global cur,con
con = None
try:
con = psycopg2.connect(database='xx', user='xx',password='xxx',host='localhost')
cur = con.cursor()
qry=" random non important query"
cur.execute(qry)
data = cur.fetchall()
for result in data:
family = result[0]
prefix = result[1]
iserver = result[2]
version = result[3]
login = result[4]
password = result[5]
service = result[6]
proxyHost = result[7]
proxyUser = result[8]
proxyPass = result[9]
parts=iserver.split(":")
host=parts[0]
port=parts[1]
if checkServer.checkit(host,port):
##SUPOSE TO AS START THREAD
if version == '7' or version == '8':
extractAdapter(env,family,iserver,login,password,prefix,proxyUser,proxyPass,proxyHost,service)
elif version == '60' or version == '61':
print "Version 6.0 and 6.1 not supported yet"
else:
print iserver+"is offline"
#TO END THREAD
except psycopg2.DatabaseError, e:
print 'Error %s' % e
sys.exit(1)
finally:
if con:
con.close()
And this is the way i call the method extract on runme.py 这就是我在runme.py上调用方法提取的方式
import extract_adapter_thread
from datetime import datetime
startTime = datetime.now()
print"------------------------------"
extract_adapter_thread.extract('TEST')
print"------------------------------"
print(datetime.now()-startTime)
By the way, the code is working just fine. 顺便说一句,代码运行正常。 no errors. 没有错误。
Threading will block very heavily within Python on non-IO bound problems because of the Global Interpreter Lock . 由于Global Interpreter Lock的存在,在非IO绑定的问题上,线程将在Python中严重阻塞。 Thus you're probably better off doing multiprocessing -- which comes with a Queue class (see this SO Link for an example of using a mp queue). 因此,您最好进行多重处理 -Queue类附带了此处理 (有关使用mp队列的示例,请参见此SO Link )。
This should let you work with many separate processes simultaneously (like batching your 5 jobs at a time out of 120). 这应该使您可以同时处理许多单独的流程(例如,一次批出5个作业,而不是120个)。 Note that the overhead of a process is higher than that of a thread, so for small tasks you'll pay a price for using multiprocessing over threading. 请注意,进程的开销比线程的开销高,因此对于小型任务,您将为在线程上使用多处理付出代价。 Your tasks sounds large enough to warrant such costs though. 您的任务听起来足够大,足以支付此类费用。
If everything is threadsafe, you could use the threading
module: 如果一切都是线程安全的,则可以使用threading
模块:
import threading
starttime=datetime.now()
print "-"*10
code=threading.thread(target=extract_adapter_thread.extract,args=['TEST'])
code.daemon=True
code.start()
print "-"*10
print(datetime.now()-starttime)
I really don't know if this will help much, but is an snippet of code that I had in my HD and well... here it goes. 我真的不知道这是否会有所帮助,但这只是我在高清机上拥有的一小段代码……好吧。 Is a basic thing to see the difference of pinging some ips in parallel or sequentially (requires linux, though). 了解并行或顺序ping某些ip的区别是一件基本的事情(不过需要linux)。 It's very simple and is not a direct answer to your specific problem but... since you said that you are new to Python , it may give you some ideas. 这很简单,不能直接解决您的特定问题,但是...由于您说过您是Python的新手 ,所以它可能会给您一些想法。
#!/usr/bin/env python
import datetime
import subprocess
import threading
ipsToPing = [
"google.com",
"stackoverflow.com",
"yahoo.com",
"terra.es",
]
def nonThreadedPinger():
start = datetime.datetime.now()
for ipToPing in ipsToPing:
print "Not-threaded ping to %s" % ipToPing
subprocess.call(["/bin/ping", "-c", "3", "-W", "1.0", ipToPing], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
end = datetime.datetime.now()
print ("Non threaded ping of %s ips took: %s." % (len(ipsToPing), end-start))
def _threadedPingerAux(ipToPing):
print "Threaded ping to %s" % ipToPing
subprocess.call(["/bin/ping", "-c", "3", "-W", "1.0", ipToPing], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
def threadedPinger():
retval = dict.fromkeys(ipsToPing, -1)
threads = list()
start = datetime.datetime.now()
for ipToPing in ipsToPing:
thread = threading.Thread(target=_threadedPingerAux, args=[ipToPing])
thread.start()
threads.append(thread)
for thread in threads:
thread.join()
end = datetime.datetime.now()
print ("Treaded ping of %s ips took: %s" % (len(ipsToPing), end-start))
if __name__ == "__main__":
threadedPinger()
nonThreadedPinger()
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.