簡體   English   中英

Python:如何在多處理中使用 class 方法?

[英]Python: How to use class methods in multiprocessing?

我有以下 class 方法來解析單個 URL:

product = Product(links[0], user_agents)
result = product.parse()

和 class 代碼:

class Product:
    soup = None
    url = None

    def __init__(self, url, user_agents):
        self.url = url
        print('Class Initiated with URL: {}'.format(url))
        # Randomize the user agent
        user_agent = get_random_user_agent(user_agents)
        user_agent = user_agent.rstrip('\n')

        if 'linux' in user_agent.lower():
            sec_ch_ua_platform = 'Linux'
        elif 'mac os x' in user_agent.lower():
            sec_ch_ua_platform = 'macOS'
        else:
            sec_ch_ua_platform = 'Windows'

        headers = {
            
        }
        r = create_request(url, None, headers=headers, is_proxy=False)
        if r is None:
            raise ValueError('Could not get data')
        html = r.text.strip()
        self.soup = BeautifulSoup(html, 'lxml')

    def parse(self):
        record = {}
        name = ''
        price = 0
        user_count_in_cart = 0
        review_count = 0
        rating = 0
        is_personalized = 'no'

        try:
            name = self.get_name()
            price = self.get_price()
            is_pick = self.get_is_pick()

現在我想在多處理中調用parse() 我該怎么做? 對於一條記錄,我這樣做:

product = Product(links[0], user_agents)
result = product.parse()

With currecnt class you may need to create function which gets url and it creates product = Product(url,...) and it runs product.parse() - and this new function you can use .map(new_function, links)

像這樣的東西:

def check(url):
    product = Product(url, user_agents)
    result = product.parse()
    return result

for multiprocessing.pool.Pool() as p:
    results = p.map(check, links)

暫無
暫無

聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.

 
粵ICP備18138465號  © 2020-2024 STACKOOM.COM