如何在 Python 中使用 BeautifulSoup 获取视频源

Question

我正在尝试在网站中找到可下载的视频链接。 例如，我正在使用这样的网址https://www.loc.gov/item/2015669100/ 。 可以看到在mejs__mediaelement div标签下有一个m3u8视频链接。

但是我的代码没有打印任何东西。 这意味着它没有找到网站的视频网址。

我的代码在下面

from bs4 import BeautifulSoup
from urllib.request import Request, urlopen

with open('pages2crawl.txt', 'r') as inFile:
        lines = [line.rstrip() for line in inFile]

for page in lines:
        req = Request(page, headers={'User-Agent': 'Mozilla/5.0'})
        soup = BeautifulSoup(urlopen(req).read(), 'html.parser')
        pages = soup.findAll('div', attrs={'class' : 'mejs__mediaelement'})
        for e in pages:
                video = e.find("video").get("src")
                if video.endswith("m3u8"):
                        print(video)

Answer 1

如果您只想制作一个简单的脚本，使用正则表达式可能会更容易。

import re, requests

s = requests.Session() #start the session
data = s.get(url) #http get request to download data
data = data.text #get the raw text

vidlinks = re.findall("src='(.*?).m3u8'/>", data) #find all between the two parts in the data
print(vidlinks[0] + ".m3u8") #print the full link with extension

Answer 2

您可以使用 CSS 选择器source[type="application/x-mpegURL"]提取 MPEG 链接（或使用source[type="video/mp4"]提取 mp4 链接）：

import requests
from bs4 import BeautifulSoup

url = "https://www.loc.gov/item/2015669100/"
soup = BeautifulSoup(requests.get(url).content, "html.parser")

link_mpeg = soup.select_one('source[type="application/x-mpegURL"]')["src"]
link_mp4 = soup.select_one('source[type="video/mp4"]')["src"]
print(link_mpeg)
print(link_mp4)

印刷：

https://tile.loc.gov/streaming-services/iiif/service:afc:afc2010039:afc2010039_crhp0001:afc2010039_crhp0001_mv04/full/full/0/full/default.m3u8
https://tile.loc.gov/storage-services/service/afc/afc2010039/afc2010039_crhp0001/afc2010039_crhp0001_mv04.mp4