django 'set' object has no attribute 'items' 错误

Question

我正在尝试在 AWS 上部署我的 django 应用程序。 我使用 bs4 和请求在我的应用程序中创建了一个网络爬虫。我用它来从电子商务网站获取数据。它在亚马逊上完美运行，但是当我尝试从 newegg 抓取时它抛出这个（参见图 1）。 但相同的代码适用于本地主机。 首先我认为这是用户代理问题，我尝试使用其他用户代理但没有用。

我真的不知道是什么问题，所以如果您知道导致此错误的原因，那么请告诉它会非常有帮助。

下面的代码和图像：

图像1

headers = {"User-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36"}
url = "https://www.newegg.com/p/pl?d="+query
data = re.get(url,headers=headers)
soup = bs4.BeautifulSoup(data.content, 'lxml')

items_container = soup.find("div", {"class":"list-wrap"})
items = items_container.find("div", {"class":"items-view"})

prdList = []
for item in items.find_all("div", {"class":"item-container"}):
    header = item.find("a", {"class":"item-title"})
    title = header.text
    link = header["href"]

    img = item.find("a", {"class":"item-img"})
    imgsrc = img.find("img")["src"]           

    price = item.find("li", {"class":"price-current"})
    if price == None:
        price = '$--'
    else:
        price = price.find("strong").text    

    prdList.append(Scraper.prdTmplt(imgsrc, title, price, link, site))

allProduct = prdList
return allProduct

错误：

Environment:


Request Method: GET
Request URL: http://ec2-54-179-183-56.ap-southeast-1.compute.amazonaws.com:8000/search?sq=gaming+laptop&site_selected=newegg&sq-submit=Search

Django Version: 2.2.6
Python Version: 3.6.8
Installed Applications:
['groups.apps.GroupsConfig',
 'product_details.apps.ProductDetailsConfig',
 'scraper.apps.ScraperConfig',
 'pages.apps.PagesConfig',
 'django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware']



Traceback:

File "/home/ubuntu/django/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner
  34.             response = get_response(request)

File "/home/ubuntu/django/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  115.                 response = self.process_exception_by_middleware(e, request)

File "/home/ubuntu/django/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  113.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/home/ubuntu/best-price-py/pages/views.py" in search
  29.             context['bb'] = Scraper.neweggList(search_query, site)

File "/home/ubuntu/best-price-py/scraper/scrapers.py" in neweggList
  120.         data = re.get(url)

File "/home/ubuntu/django/lib/python3.6/site-packages/requests/api.py" in get
  75.     return request('get', url, params=params, **kwargs)

File "/home/ubuntu/django/lib/python3.6/site-packages/requests/api.py" in request
  60.         return session.request(method=method, url=url, **kwargs)

File "/home/ubuntu/django/lib/python3.6/site-packages/requests/sessions.py" in request
  519.         prep = self.prepare_request(req)

File "/home/ubuntu/django/lib/python3.6/site-packages/requests/sessions.py" in prepare_request
  462.             hooks=merge_hooks(request.hooks, self.hooks),

File "/home/ubuntu/django/lib/python3.6/site-packages/requests/models.py" in prepare
  314.         self.prepare_headers(headers)

File "/home/ubuntu/django/lib/python3.6/site-packages/requests/models.py" in prepare_headers
  446.             for header in headers.items():

Exception Type: AttributeError at /search
Exception Value: 'set' object has no attribute 'items'

Answer 1

您传递的标题字典被呈现为一组而不是字典。 查看图像中的转储。