[英]How to compare between two dictionaries using threads
Im currently working on a comparison where I am trying to solve on how I am able to compare between two dictionaries where the first requests does a GET and scrapes the data to a dictionary and then I want to compare to for the next request using the same method and see if there has been any changes on the webpage.我目前正在进行比较,我试图解决我如何能够在两个字典之间进行比较,其中第一个请求执行 GET 并将数据刮到字典,然后我想比较下一个使用相同的请求方法,看看网页是否有任何变化。 I have currently done:
我目前已经完成:
import random
import threading
import time
from concurrent.futures import as_completed
from concurrent.futures.thread import ThreadPoolExecutor
import requests
from bs4 import BeautifulSoup
URLS = [
'https://github.com/search?q=hello+world',
'https://github.com/search?q=python+3',
'https://github.com/search?q=world',
'https://github.com/search?q=i+love+python',
'https://github.com/search?q=sport+today',
'https://github.com/search?q=how+to+code',
'https://github.com/search?q=banana',
'https://github.com/search?q=android+vs+iphone',
'https://github.com/search?q=please+help+me',
'https://github.com/search?q=batman',
]
def doRequest(url):
response = requests.get(url)
time.sleep(random.randint(10, 30))
return response, url
def doScrape(response):
soup = BeautifulSoup(response.text, 'html.parser')
return {
'title': soup.find("input", {"name": "q"})['value'],
'repo_count': soup.find("span", {"data-search-type": "Repositories"}).text.strip()
}
def checkDifference(parsed, url):
def threadPoolLoop():
with ThreadPoolExecutor(max_workers=1) as executor:
future_tasks = [
executor.submit(
doRequest,
url
) for url in URLS]
for future in as_completed(future_tasks):
response, url = future.result()
if response.status_code == 200:
checkDifference(doScrape(response), url)
while True:
t = threading.Thread(target=threadPoolLoop, )
t.start()
print('Joining thread and waiting for it to finish...')
t.join()
My problem is that I do not know how I can print out whenever there has been a change for either title or/and repo_count?我的问题是,当标题或/和 repo_count 发生更改时,我不知道如何打印出来? (The whole point will be that I will run this script 24/7 and I always want it to print out whenever there has been a change)
(重点是我将 24/7 运行这个脚本,并且我总是希望它在有变化时打印出来)
If you're looking for a simple method to compare two dictionaries, there are a few different options.如果您正在寻找一种简单的方法来比较两个字典,那么有几个不同的选项。
Some good resources to begin:一些好的资源开始:
Let's start with two dictionaries to compare 👇 Some added elements, some removed, some changed, some same.让我们从两个字典开始比较👇 一些添加的元素,一些删除的,一些更改的,一些相同的。
dict1 = {
"value_2": 2,
"value_3": 3,
"value_4": 4,
"value_5": "five",
"value_6": "six",
}
dict2 = {
"value_1": 1,
"value_2": 2,
"value_4": 4
}
You could probably use the unittest
library.您可能可以使用
unittest
库。 Like this:像这样:
>>> from unittest import TestCase
>>> TestCase().assertDictEqual(dict1, dict1) # <-- No output, because they are the same
>>> TestCase().assertDictEqual(dict1, dict2) # <-- Will raise error and display elements which are different
AssertionError: {'value_2': 2, 'value_3': 3, 'value_4': 4, 'value_5': 'five', 'value_6': 'six'} != {'value_1': 1, 'value_2': 3, 'value_4': 4}
- {'value_2': 2, 'value_3': 3, 'value_4': 4, 'value_5': 'five', 'value_6': 'six'}
+ {'value_1': 1, 'value_2': 3, 'value_4': 4}
But the challenge there is that it will raise an error when they are different;但挑战在于,当它们不同时会引发错误; which is probably not what you're looking for.
这可能不是你要找的。 You simply want to see when they are different.
您只是想看看它们何时不同。
Another method is the deepdiff
library.另一种方法是
deepdiff
库。 Like this:像这样:
>>> from deepdiff import DeepDiff
>>> from pprint import pprint
>>> pprint(DeepDiff(dict1, dict2))
{'dictionary_item_added': [root['value_1']],
'dictionary_item_removed': [root['value_3'], root['value_5'], root['value_6']],
'values_changed': {"root['value_2']": {'new_value': 3, 'old_value': 2}}}
Or, you could easily craft your own functions.或者,您可以轻松地制作自己的函数。 Like this 👇 (functions copied from here )
像这样👇(从这里复制的功能)
>>> from pprint import pprint
>>> def compare_dict(d1, d2):
... return {k: d1[k] for k in d1 if k in d2 and d1[k] == d2[k]}
>>> pprint(compare_dict(dict1, dict2))
{'value_4': 4}
>>> def dict_compare(d1, d2):
... d1_keys = set(d1.keys())
... d2_keys = set(d2.keys())
... shared_keys = d1_keys.intersection(d2_keys)
... added = d1_keys - d2_keys
... removed = d2_keys - d1_keys
... modified = {o: {"old": d1[o], "new": d2[o]} for o in shared_keys if d1[o] != d2[o]}
... same = set(o for o in shared_keys if d1[o] == d2[o])
... return {"added": added, "removed": removed, "modified": modified, "same": same}
>>> pprint(dict_compare(dict1, dict2))
{'added': {'value_6', 'value_3', 'value_5'},
'modified': {'value_2': {'old': 2, 'new': 3}},
'removed': {'value_1'},
'same': {'value_4'}}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.