获取请求状态码 200 响应 401 错误

Question

I want to get a request from the api of this website https://www.flashscore.com/match/tE4RoHzB/#match-summary/match-statistics/0 I want to request this url to scraping data https://d.flashscore.com/x/feed/df_st_1_tE4RoHzB The Status Code is 200 but when I run the code I got 401

import requests

url = "https://d.flashscore.com/x/feed/df_st_1_tE4RoHzB"

response = requests.get(url)

print(response.status_code)

How can I fix that?

Answer 1

To get that html for that particular url, you need to pass in the 'x-fsign' in the headers, which is found in the <script> tags data in the root/main url.

import requests
from bs4 import BeautifulSoup
import json
import re


url = 'https://www.flashscore.com/match/tE4RoHzB/'
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36'}

response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.text, 'html.parser')

scripts = soup.find_all('script',{'type':"text/javascript"})
for script in scripts:
    if 'window.environment =' in str(script):
        scriptStr = str(script)
        jsonMatch = re.compile("{.*}")
        jsonStr = jsonMatch.search(scriptStr)[0]
        jsonData = json.loads(jsonStr)

fsign = jsonData['config']['app']['feed_sign']
headers.update({'x-fsign':fsign})
url = "https://d.flashscore.com/x/feed/df_st_1_tE4RoHzB"


response = requests.get(url, headers=headers)

print(response.status_code)
print(response.text)

Output:

200
SE÷Match¬~SG÷Ball Possession¬SH÷41%¬SI÷59%¬~SG÷Goal Attempts¬SH÷10¬SI÷20¬~SG÷Shots on Goal¬SH÷4¬SI÷3¬~SG÷Shots off Goal¬SH÷3¬SI÷9¬~SG÷Blocked Shots¬SH÷3¬SI÷8¬~SG÷Free Kicks¬SH÷8¬SI÷11¬~SG÷Corner Kicks¬SH÷6¬SI÷7¬~SG÷Offsides¬SH÷1¬SI÷1¬~SG÷Throw-in¬SH÷15¬SI÷15¬~SG÷Goalkeeper Saves¬SH÷0¬SI÷4¬~SG÷Fouls¬SH÷10¬SI÷7¬~SG÷Total Passes¬SH÷389¬SI÷584¬~SG÷Tackles¬SH÷14¬SI÷15¬~SG÷Attacks¬SH÷107¬SI÷105¬~SG÷Dangerous Attacks¬SH÷77¬SI÷53¬~SE÷1st Half¬~SG÷Ball Possession¬SH÷37%¬SI÷63%¬~SG÷Goal Attempts¬SH÷5¬SI÷13¬~SG÷Shots on Goal¬SH÷1¬SI÷1¬~SG÷Shots off Goal¬SH÷2¬SI÷7¬~SG÷Blocked Shots¬SH÷2¬SI÷5¬~SG÷Free Kicks¬SH÷2¬SI÷5¬~SG÷Corner Kicks¬SH÷2¬SI÷2¬~SG÷Offsides¬SH÷0¬SI÷0¬~SG÷Throw-in¬SH÷10¬SI÷9¬~SG÷Goalkeeper Saves¬SH÷0¬SI÷1¬~SG÷Fouls¬SH÷5¬SI÷2¬~SG÷Total Passes¬SH÷188¬SI÷331¬~SG÷Tackles¬SH÷8¬SI÷10¬~SG÷Attacks¬SH÷50¬SI÷61¬~SG÷Dangerous Attacks¬SH÷35¬SI÷29¬~SE÷2nd Half¬~SG÷Ball Possession¬SH÷45%¬SI÷55%¬~SG÷Goal Attempts¬SH÷5¬SI÷7¬~SG÷Shots on Goal¬SH÷3¬SI÷2¬~SG÷Shots off Goal¬SH÷1¬SI÷2¬~SG÷Blocked Shots¬SH÷1¬SI÷3¬~SG÷Free Kicks¬SH÷6¬SI÷6¬~SG÷Corner Kicks¬SH÷4¬SI÷5¬~SG÷Offsides¬SH÷1¬SI÷1¬~SG÷Throw-in¬SH÷5¬SI÷6¬~SG÷Goalkeeper Saves¬SH÷0¬SI÷3¬~SG÷Fouls¬SH÷5¬SI÷5¬~SG÷Total Passes¬SH÷201¬SI÷253¬~SG÷Tackles¬SH÷6¬SI÷5¬~SG÷Attacks¬SH÷57¬SI÷44¬~SG÷Dangerous Attacks¬SH÷42¬SI÷24¬~A1÷¬~