有没有办法从网站上的shadow-root中提取信息？

Question

I am setting up code to check the reputation of any URL Eg http://go.mobisla.com/ on Website " https://www.virustotal.com/gui/home/url "

First, the very basic thing I am doing is to extract all the Website contents using BeautifulSoup but seems the information I am looking for is in shadow-root(open) -- div.detections and span.individual-detection.

Example Copied Element from Webpage results:

No engines detected this URL

I am new to Python, wondering if you can share the best way to extract the information

Tried requests.get() function but it doesn't give the required information

 import requests import os,sys from bs4 import BeautifulSoup import pandas as pd url_check = "deloplen.com:443" url = "https://www.virustotal.com/gui/home/url" req = requests.get(url + url_str) html = req.text soup = BeautifulSoup(html, 'html.parser') print(soup.prettify()) 

Expect to see "2 engines detected this URL" along with Detection Example: Dr. Web Malicious

Answer 1

If you use their website, it'll only return a loading screen for VirusTotal, as this isn't the proper way.

What Shows Up:

Instead, what you're supposed to do is use their public API to make requests. However, you'll have to make an account to obtain a Public API Key.

You can use this code which is able to retrieve JSON info about the link. However, you'll have to fill in the API KEY with yours.

import requests, json

user_api_key = "<api key>"
resource = "deloplen.com:443"

# feel free to remove this, just makes it look nicer
def pp_json(json_thing, sort=True, indents=4):
    if type(json_thing) is str:
        print(json.dumps(json.loads(json_thing), sort_keys=sort, indent=indents))
    else:
        print(json.dumps(json_thing, sort_keys=sort, indent=indents))
        return None

response = requests.get("https://www.virustotal.com/vtapi/v2/url/report?apikey=" + user_api_key + "&resource=" + resource)

json_response = response.json()

pretty_json = pp_json(json_response)

print(pretty_json)

If you want to learn more about the API, you can use their documentation .