为什么我看不到我请求的页面？

Question

I've been learning Python and tried Web Scraping. I could manage to scrape Google Result Page for a normal Google Search, though the page was depreciated idk why. Tried the same for Google Images, and it is depreciated as well. It doesn't appear the same as it was appearing in the browser.

Here 's my code.

from bs4 import BeautifulSoup
import requests
from PIL import Image
from io import BytesIO

search = input("Search for : ")
params = {"tbm": "isch", "source": "hp", "q": search}
r = requests.get("https://www.google.com/search", params=params)
print("URL :", r.url)
print("Status : ", r.status_code, "\n\n")

f = open("ImageResult.html", "w+")
f.write(r.text)

For example, I search for "Goku". The Google Image returns this page.

When I click on the first image, a popup opens. Or say I press ctrl+click. I reach this page.

On this page I can see that the actual image's URL can be accessed through maybe the current url or the link at the "View Image" button. But the issue is, I can't reach this page/popup in the version of the page that I am able to get when I request this page.

UPDATE : I'm sharing the page I am getting.

Answer 1

This depends on a lot of factors like user agent string , cookies and also google experiments . Google is known for serving different ways of same content for many users.On search ,Google loads different pages based on site speed and user agent.Google also randomly runs experiments on searchpage design,etc before rollng in public to implement A/B testing dynamically.

Answer 2

Google Organic results have very little JavaScript and you still can parse data from the <script> tags.

Besides that, the most often problem why you don't see the same results as in your browser is because there's no user-agent being passed into request headers thus when no user-agent is specified while using requests library, it defaults to python-requests and Google understands that it's a bot/script, then it blocks a request ( or whatever it does ) and you receive a different HTML with different CSS selectors.Check what's your user-agent .

Pass user-agent :

headers = {
    'User-agent':
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582'
}

requests.get('URL', headers=headers)

Alternatively, you can achieve the same thing by using Google Organic Results API from SerpApi. It's a paid API with a free plan.

The difference in your case is that you don't have to spend time trying to bypass blocks from Google and figuring out why certain things don't work as they should, and you don't have to maintain the parser over time.

Very simple example code to integrate:

import os
from serpapi import GoogleSearch

params = {
  "engine": "google",
  "q": "how to create minecraft server",
  "hl": "en",
  "gl": "us",
  "api_key": os.getenv("API_KEY"),
}

search = GoogleSearch(params)
results = search.get_dict()

for result in results["organic_results"]:
  print(result["link"], sep="\n")

----------
'''
https://help.minecraft.net/hc/en-us/articles/360058525452-How-to-Setup-a-Minecraft-Java-Edition-Server
https://www.minecraft.net/en-us/download/server
https://www.idtech.com/blog/creating-minecraft-server
# other results
'''

Disclaimer, I work for SerpApi.