如何使用 beatifulsoup 从 lxml 数据字符串中过滤出带有空格的正确单词

Question

hi guyz here i'm getting on string that contains lots of html data (in single string)

from bs4 import BeautifulSoup
import requests
import bs4
url = "any randome url"
html = requests.get(url).text
soup = BeautifulSoup(html,'lxml')
web_page=soup.get_text().strip()
print(web_page.lower())

and some of the words are coming in output like

conditionstravel for conditions & travel

vaccinationstreatment for vaccination & treatment

the web page is scraping is correct, but this is not expected,

bcoz some of the tags are ending with text conditions and next tag is starting text with travels so that's why it's coming like conditionstravel

here i'm willing scrape the web page by one by one tags and make it as a web_page_data_list so is there any way to scrape all tags texts with separate state like above

and the problem is we can't give specific dictionary words for this is that possible with beautiful soup or any other package will help to extract this properly.?

Answer 1

Use separator=' ' parameter in .get_text() method. Also you can supply strip=True to strip whitespace characters automatically of every separated word.

For example:

import bs4
from bs4 import BeautifulSoup

txt = '''<div>Hello<span>World</span></div>'''

soup = BeautifulSoup(txt, 'html.parser')
web_page=soup.get_text(strip=True, separator=' ')
print(web_page.lower())
print(bs4.__version__)

Prints:

hello world
4.9.1