我无法使用 python selenium 下载谷歌图片

Question

Hi I'm crawling a google image using selenium. But it didn't work well. How can I get this code to work? My code is like below.

Previously, I used google_images_download and suddenly got stuck. So I'm looking for a new way and I hope someone can help Thank you

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import json
import os
import urllib.request as urllib2
import argparse

searchterm = 'spider' # will also be the name of the folder
url = "https://www.google.co.in/search?q="+searchterm+"&source=lnms&tbm=isch"
# NEED TO DOWNLOAD CHROMEDRIVER, insert path to chromedriver inside parentheses in following line
browser = webdriver.Chrome('C:\Python27\Scripts\chromedriver')
browser.get(url)
header={'User-Agent':"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36"}
counter = 0
succounter = 0

if not os.path.exists(searchterm):
    os.mkdir(searchterm)

for _ in range(500):
    browser.execute_script("window.scrollBy(0,10000)")

for x in browser.find_elements_by_xpath('//div[contains(@class,"rg_meta")]'):
    counter = counter + 1
    print("Total Count:", counter)
    print("Succsessful Count:", succounter)
    print("URL:",json.loads(x.get_attribute('innerHTML'))["ou"])

    img = json.loads(x.get_attribute('innerHTML'))["ou"]
    imgtype = json.loads(x.get_attribute('innerHTML'))["ity"]
    try:
        req = urllib2.Request(img, headers={'User-Agent': header})
        raw_img = urllib2.urlopen(req).read()
        File = open(os.path.join(searchterm , searchterm + "_" + str(counter) + "." + imgtype), "wb")
        File.write(raw_img)
        File.close()
        succounter = succounter + 1
    except:
            print("can't get img")

print (succounter, "pictures succesfully downloaded")
browser.close()

Answer 1

我决定从 Google 图片以外的其他网站抓取图片。

Answer 2

I also faced problem with crawling images from google like your method as using rg_meta

google image search result webpage source code has been changed and they don't give rg_meta anymore since the beginning of 2020.

rg_meta tag also changed to randomly string like

rg_X XXXXXX XXXXXX

I think Google set about to ban crawling bots and leading to use Google Custom Search APIs.