网页抓取图像 URL 返回为 ''

Question

I think my issue is with javascript running on the page and not loading the images until i scroll down. could anyone help me out with this? the script works fine until i hit the "ZendikarRising(ZNR)" where there are a lot more images on the page. i'm then given Failed to save imageMakindi Ox (ZNR).png from url... it should say a URL but it's returning '' there are some DEBUG codes i have incorporated to bypass missing card URLs but I'm missing tons.

I've tried removing empty fields but if you run it you can see i have even number of card names and URLs (some of which are blank) so removing the empty URLs would throw off the totals and would result in me missing cards from the set.

Here is the code in question

import requests
import os
from os.path import basename
from bs4 import BeautifulSoup
 
path = os.getcwd()
print ("The current working directory is %s" % path)
 
url = 'https://scryfall.com/sets'
r=requests.get(url).text
soup = BeautifulSoup(r, 'html.parser')
 
####################GATHERS ALL URLS FROM SET DIRECTORY#####################
links = []
Urls = []
for link in soup.findAll('a'):
    links.append(link.get('href'))
 
for link in links:
    if link != None:
        if 'https://scryfall.com/sets/' in link:
            if link not in Urls:
                Urls.append(link)
 
#################START OF ALL URL LOOPS################################
for Url in Urls: ##goes threw all the URLS gathered from the sets links
    r=requests.get(Url).text
    soup = BeautifulSoup(r, 'html.parser')
 
    temp = soup.find('h1', {'class': 'set-header-title-h1'}).contents
    temp = ''.join(temp)
    temp = temp.strip()
    temp = temp.replace(':', '')
    temp = temp.replace(' ', '')
 
    test2 = (f"{path}\\{temp}")
#############################################MAKE DIRECTORY FOR SET FOLDERS##################
    try:
        os.mkdir(test2)
    except OSError:
        print ("Creation of the directory %s failed" % test2)
    else:
        print ("Successfully created the directory %s " % test2)
 
############################################GATHER ALL IMAGES####################
    images = soup.find_all('img')
 
    pictures = [] ##stores all the picture URLS
    names = [] ##stores all the name
 
    for image in images[:-1]:
        names.append(image.get('alt'))
        pictures.append(image.get('src'))
####################SAVES ALL IMAGES AS FILES#################
 
    x=0
    for i in pictures:
        fn = names[x] + '.png'
        try:
            with open(f'{test2}\\'+basename(fn),"wb") as f:
 
                f.write(requests.get(i).content)
                f.close
                ##print(i)
                ##print(f'saved {fn} to {path}')
                x+=1
        except OSError:
            print(f"Failed to save image{fn} from url{i}")
            print(len(pictures))
            print(len(names))
            exit()
##################RESETS IMAGES AND NAMES FOR NEXT SET FOLDER#############
 
    pictures.clear()
    names.clear()
Print("Completed With No Errors")

Answer 1

Indeed the images are lazy loaded by a JS script and though you find no <img> tags with src attributes later in the page.

However the solution is pretty simple. If you look at several <img> tags that are not loaded, you will see that the image link is not present in the src attribute, but rather in the data-src attribute.

For example:

<img alt="Wayward Guide-Beast (ZNR)" class="card znr border-black" data-component="lazy-image" data-src="https://c1.scryfall.com/file/scryfall-cards/normal/front/e/b/ebfe94fc-7a98-4f53-8fd0-f5fd016b1873.jpg?1599472001" src="" title="Wayward Guide-Beast (ZNR)"/>

So all you have to do is check whether src is empty and if so scrape the data-src attribute.