Python 像浏览器一样请求？

Question

I want to get a webdocument from 'https://www.fanfiction.net/s/5218118/1/', but I am sadly unable to replicate the behaviour of my browser - the server always sends me something along the lines of "please enable cookies" or "complete Captcha". Is there a way to send requests like a browser, so the server delivers me the same document as if I am a browser? I already googled and tried to integrate cookies and a fake Useragent. Here is my code:

import requests
from fake_useragent import UserAgent

url = 'https://www.fanfiction.net/s/5218118/1/'

ua = UserAgent()
S = requests.Session()

header = {'User-Agent':str(ua.chrome)}
res = S.get(url, headers=header)
cookies = dict(res.cookies)


response = S.get(url, headers=header, cookies=cookies)

Thanks already in advance: EDIT, I know that I could use selenium, but I do not want to always update my chromedriver. and also I do not want to waste performance on selenium.

Answer 1

Saw your EDIT but just in case, ...

Simple example with selenium, that give you the storytext

from selenium import webdriver
from bs4 import BeautifulSoup


browser = webdriver.Chrome('C:\Program Files\ChromeDriver\chromedriver.exe')
browser.get('https://www.fanfiction.net/s/5218118/1/')

soup=BeautifulSoup(browser.page_source, 'lxml')

print(soup.select_one('#storytext').get_text())

browser.close()

Edit

Edited based on your question and the fact that the site is protected by cloudflare to avoid ddos attacks.

You could extract the tag texts by selenium, but as in example above, I use beautifulsoup

You are right, inspect the html with developer tools and the tag part looks like this:

 <span class="xgray xcontrast_txt"> Rated: <a class="xcontrast_txt" href="https://www.fictionratings.com/" target="rating">Fiction T</a> - English - Romance/Adventure - Naruto U., Hinata H. - Chapters: 6 - Words: 14,894 - Reviews: <a href="/r/13747729/">5</a> - Favs: 29 - Follows: 24 - Updated: <span data-xutime="1610096566">33m ago</span> - Published: <span data-xutime="1605552788">Nov 16, 2020</span> - id: 13747729 </span>

A span with the classes xgray xcontrast_txt so we select it like that:

tags = soup.select_one('span.xgray.xcontrast_txt').get_text(strip=True)

You may wanna know more about beautifulsoup?

Example

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from bs4 import BeautifulSoup


browser = webdriver.Chrome('C:\Program Files\ChromeDriver\chromedriver.exe')
browser.get('https://www.fanfiction.net/s/5218118/4/Yet-again-with-a-little-extra-help')

try:
    # wait until certain element with id 'storytext' showed up
    element = WebDriverWait(browser, 10).until(
        EC.presence_of_element_located((By.ID, 'storytext'))
    )
    
    soup=BeautifulSoup(browser.page_source, 'lxml')

    storytext = soup.select_one('#storytext').get_text()
    tags = soup.select_one('span.xgray.xcontrast_txt').get_text(strip=True)
    
    print(tags)
    print(storytext)
    
finally:
    browser.close()