如何在没有看到要刮擦的代码的网页中以刮擦的方式获取数据

Question

I'm trying to get the names of the users and the content of the comments that exist on this page :

User and text that I need to extract:

When I test the extraction with the chrome plugin Xpath helper , I am getting the user names with the statement:

//*[@id="livefyre"]/div/div/div/div/article/div/header/a/span

and the comments, I get them with:

//*[@id="livefyre"]/div/div/div/div/article/div/section/div/p

When I do the test in the scrapy console, with the query:

response.xpath(//*[@id="livefyre"]/div/div/div/div/article/div/section/div/p).extract()

I get a [] ;

I've also tried with:

response.xpath (//*[@id="livefyre"]/div/div/div/div/article/div/section/div/p.text()).extract()

The same thing happens with my code.

Verifying the code of the page, I see that all those comments do not exist in the html code.

When I inspect the page, for example, I see the comment text:

But when, I check the html code of the page I do not see anything :

Where am I making a mistake?

Thanks for help.

Answer 1

As you stated, there isn't any comment in the code of page, that mean website is being rendered through javascript, There are two ways you can scrap these kind of websites

First,

use scrapy-splash to render javascript

second,

find the api/network call that brings the comments, mock that request in scrapy to get your data.