如何从简体中文网站抓取内容？

Question

I have tested this code on various English language websites with no problem. However, when I tried to scrape content from a website that's in Simplified Chinese, the data appeared as gibberish in the CSV file. In addition, the body of the article was spread out over multiple rows in Excel, not contained in one cell. Can someone help?

install.packages("rvest")
library(rvest)

###specifying the URL for the website you want to scrape ###
url <-'https://new.qq.com/omn/20190823/20190823A02W4Q00.html'

##reading the HTML code from the website
webpage <- read_html(url)

###using CSS selectors to scrape the title
title_html <- html_nodes(webpage,'h1')

###Converting the main text data to text
title_data <- html_text(title_html)

###using CSS selectors to scrape the body
text_html <- html_nodes(webpage,'.one-p')

###Converting the body data to text
text_data <- html_text(text_html)


d <- data.frame(text_data)
write.csv(d,"chinesetext.csv")

Answer 1

Most of these problems are caused encoding. I try guess_encoding function. And it guessed UTF-8 encoding. But it's not working. You can see this code.

Error in doc_parse_raw(x, encoding = encoding, base_url = base_url, as_html = as_html,  : 
input conversion failed due to input error, bytes 0xC8 0xDD 0x2D 0x2D [6003]

So i change using Extended Unix Code. It's working.

url <-'https://new.qq.com/omn/20190823/20190823A02W4Q00.html'
webpage <- read_html(url, encoding="euc-cn")
title_html <- html_nodes(webpage,'h1')
title_data <- html_text(title_html)
title_data
[1] "“六稳”政策显效 抗压能力增强"

Perhaps, you want to transform data frame in chinese language. Before your code, add this code. Then you can see chinese language in global environment.

Sys.setlocale("LC_ALL", "Chinese")