网络抓取后无法从字典中检索值

Question

I was hoping people on here would be able to answer what I believe to be a simple question. I'm a complete newbie and have been attempting to create an image webscraper from the site Archdaily. Below is my code so far after numerous attempts to debug it:

#### - Webscraping 0.1 alpha -
#### - Archdaily - 

import requests
from bs4 import BeautifulSoup

# Enter the URL of the webpage you want to download the images from
page = 'https://www.archdaily.com/63267/ad-classics-house-vi-peter-eisenman/5037e0ec28ba0d599b000190-ad-classics-house-vi-peter-eisenman-image'

# Returns the webpage source code under page_doc
result = requests.get(page)
page_doc = result.content

# Returns the source code as BeautifulSoup object, as nested data structure
soup = BeautifulSoup(page_doc, 'html.parser')
img = soup.find('div', class_='afd-gal-items')
img_list = img.attrs['data-images']
for k, v in img_list():
    if k == 'url_large':
        print(v)

These elements here:

img = soup.find('div', class_='afd-gal-items')
img_list = img.attrs['data-images']

Attempts to isolate the data-images attribute, shown here:

My github upload of this portion, very long

As you can see, or maybe I'm completely wrong here, my attempts to call the 'url_large' values from this final dictionary list comes to a TypeError, shown below:

Traceback (most recent call last):
  File "D:/Python/Programs/Webscraper/Webscraping v0.2alpha.py", line 23, in <module>
    for k, v in img_list():
TypeError: 'str' object is not callable

I believe my error lies in the resulting isolation of 'data-images', which to me looks like a dict within a list, as they're wrapped by brackets and curly braces. I'm completely out of my element here because I basically jumped into this project blind (haven't even read past chapter 4 of Guttag's book yet).

I also looked everywhere for ideas and tried to mimic what I found. I've found solutions others have offered previously to change the data to JSON data, so I found the code below:

jsonData = json.loads(img.attrs['data-images'])
print(jsonData['url_large'])

But that was a bust, shown here:

Traceback (most recent call last):
  File "D:/Python/Programs/Webscraper/Webscraping v0.2alpha.py", line 29, in <module>
    print(jsonData['url_large'])
TypeError: list indices must be integers or slices, not str

There is a step I'm missing here in changing these string values, but I'm not sure where I could change them. I'm hoping someone can help me resolve this issue, thanks!

Answer 1

It's all about the types.

img_list is actually not a list, but a string. You try to call it by img_list() which results in an error.

You had the right idea of turning it into a dictionary using json.loads . The error here is pretty straight forward - jsonData is a list, not a dictionary. You have more than one image.

You can loop through the list. Each item in the list is a dictionary, and you'll be able to find the url_large attribute in each dictionary in the list:

images_json = img.attrs['data-images']
for image_properties in json.loads(images_json):
    print(image_properties['url_large'])

Answer 2

@infinity & @simic0de are both right, but I wanted to more explicitly address what I see in your code as well.

In this particular block:

img_list = img.attrs['data-images'] for k, v in img_list(): if k == 'url_large': print(v)

There is a couple syntax errors. If 'img_list' truly WAS a dictionary, you cannot iterate through it this way. You would need to use img_list.items() (for python3) or img_list.iteritems() (python2) in the second line.

When you use the parenthesis like that, it implies that you're calling a function. But here, you're trying to iterate through a dictionary. That is why you get the 'is not callable' error.

The other main issue is the Type issue. simic0de & Infinity address that, but ultimately you need to check the type of img_list and convert it as needed so you can iterate through it.

Answer 3

Source of error: img_list is a string. You have to convert it to list using json.loads and it not becomes a list of dicts that you have to loop over.

Working Solution:

import json
import requests
from bs4 import BeautifulSoup

# Enter the URL of the webpage you want to download the images from
page = 'https://www.archdaily.com/63267/ad-classics-house-vi-peter-eisenman/5037e0ec28ba0d599b000190-ad-classics-house-vi-peter-eisenman-image'

# Returns the webpage source code under page_doc
result = requests.get(page)
page_doc = result.content

# Returns the source code as BeautifulSoup object, as nested data structure
soup = BeautifulSoup(page_doc, 'html.parser')
img = soup.find('div', class_='afd-gal-items')
img_list = img.attrs['data-images']
for img in json.loads(img_list):
    for k, v in img.items():
        if k == 'url_large':
            print(v)