Detecting the language of a text from a crawled website in Python
Question
I wrote several different spiders for different websites that output the text of articles and the URLs. Example:
import scrapy
import re
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
from bs4 import BeautifulSoup
stop_words = set(stopwords.words("german"))
class FruehstueckSpider(scrapy.Spider):
name = "fruestueckerinnen"
def start_requests(self):
urls = [
'https://www.diefruehstueckerinnen.at/stadt/wien/',
]
urls += [urls[0] + 'page/' + str(i) + '/' for i in range(1,17)]
for url in urls:
yield scrapy.Request(url=url, callback=self.parse)
def parse(self, response):
hrefs = response.css('div.text > a')
yield from response.follow_all(hrefs, callback = self.parse_attr)
def parse_attr(self, response):
yield {
'text': ' '.join([i for i in word_tokenize(re.sub(pattern='[^a-zA-Z_\-ÖöÜüÄäßèé]',string= BeautifulSoup(response.css('.content-inner.single-content').get(),"html.parser").find(class_="content-inner single-content").text , repl=' ')) if i not in stop_words and not re.match('[0-9]', i) and len(i) >1]),
'url': response.request.url,
}
I want to detect the language the whole text is written into. Does it make sense to write it under 'text' and 'url' as another property? I know that there is a function called detect (and the input is a string) from langdetect , but how do I use it in this case?
2 answers
solution1
1 2020-04-20 10:50:53
You can add another field inside yield like this
from langdetect import detect # add this to your import
# change the parse_attr function like this
def parse_attr(self, response):
text = ' '.join([i for i in word_tokenize(re.sub(pattern='[^a-zA-Z_\-ÖöÜüÄäßèé]',string= BeautifulSoup(response.css('.content-inner.single-content').get(),"html.parser").find(class_="content-inner single-content").text , repl=' ')) if i not in stop_words and not re.match('[0-9]', i) and len(i) >1])
language = detect(text)
yield {
'text': text,
'language': language,
'url': response.request.url,
}
solution2
0 2020-04-20 10:36:33
The "lang"-attribute a html attribute that should define the language of a page. I suggest that you use this as a reference for the site because it is the most direct way to recognize this attribute. This attribute was defined to help for example voice software to choose the right language for the pronunciation.
<html lang="en">... </html>Adding the language to the output is a matter of taste. It doesn't hurt but do you really need it? You can always include but not use the value everywhere.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.