[英]Getting an element via LXML by XPATH
I am coding a Discord bot in a library for python, discord.py.我正在为 python、discord.py 的库中编写 Discord 机器人。
I don't need help with that but with scraping some info from the site.我不需要帮助,但需要从网站上抓取一些信息。
@commands.command(aliases=["rubyuserinfo"])
async def rubyinfo(self, ctx, input):
HEADERS = {
'User-Agent' : 'Magic Browser'
}
url = f'https://rubyrealms.com/user/{input}/'
async with aiohttp.request("GET", url, headers=HEADERS) as response:
if response.status == 200:
print("Site is working!")
content = await response.text()
soup = BeautifulSoup(content, "html.parser")
page = requests.get(url)
tree = html.fromstring(page.content)
stuff = tree.xpath('/html/body/div[4]/div/div[3]/div[3]/div/div[2]/div[1]/div[2]/div/p')
print(stuff)
else:
print(f"The request was invalid\nStatus code: {response.status}")
The website I am looking for is "https://rubyrealms.com/user/{input}/" where input is given while running h!rubyinfo USERNAME changing the link to https://rubyrealms.com/user/username/ .我正在寻找的网站是“https://rubyrealms.com/user/{input}/”,其中在运行 h!rubyinfo USERNAME 时给出输入,将链接更改为https://rubyrealms.com/user/username/ 。
On the website what I want to get is their BIO which has an XPATH of在网站上我想得到的是他们的 BIO,它有一个 XPATH
"//*[@id="content-wrap"]/div[3]/div[3]/div/div[2]/div[1]/div[2]/div/p"
where the element is:其中元素是:
<p class="margin-none font-color">
Hey! My name is KOMKO190, you maybe know me from the forums or discord. I am a programmer, I know a bit of JavaScript, small portion of C++, Python and html/css. Mostly python. My user ID is 7364. ||| 5th owner of Space Helmet :) </p>
Any help on how I would scrape that?关于我如何刮掉它的任何帮助? The only response my bot gives is "[]"我的机器人给出的唯一响应是“[]”
How about the following, using.select() method下面怎么样,使用.select() 方法
from bs4 import BeautifulSoup
html = '<p class="margin-none font-color">Hey! My name is KOMKO190 :) </p>'
soup = BeautifulSoup(html, features="lxml")
element = soup.select('p.margin-none')[0]
print(element.text)
Prints out打印出来
Hey: My name is KOMKO190 :)
Change your XPath expression for a relative one:将您的 XPath 表达式更改为相对表达式:
from lxml import html
import requests
page = requests.get('https://www.rubyrealms.com/user/KOMKO190/')
tree = html.fromstring(page.content)
stuff = tree.xpath('normalize-space(//h3[.="Bio"]/following-sibling::p/text())')
print (stuff)
Output: Output:
Hey! My name is KOMKO190, you maybe know me from the forums or discord. I am a programmer, I know a bit of JavaScript, small portion of C++, Python and html/css. Mostly python. My user ID is 7364. ||| 5th owner of Space Helmet :)
from bs4 import BeautifulSoup as bs
url = 'https://rubyrealms.com/user/username/'
session = requests.Session()
request = session.get(url=url)
if request.status_code == 200:
soup = bs(request.text, 'lxml')
print(soup.find('p', class_='margin-none font-color').text)
else:
print(request.status_code)
You need install你需要安装
pip install lxml
pip install beautifulsoup4
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.