提交表单并加载数据后抓取网站

Question

I have to scrape a website which i've reviewed and i realised that i don't need to submit any form. I have the needed urls to get the data. I'm using NodeJs and Phantom .

My problems source is something related with the session or cookies (i think).

In my web browser i can enter in this link https://www.infosubvenciones.es/bdnstrans/GE/es/convocatorias , hit on the form blue button with text "Procesar consulta". The table below will be filled. In dev tools on network tab you can see a XHR request with a link similar to https://www.infosubvenciones.es/bdnstrans/busqueda?type=convs&_search=false&nd=1594848133517&rows=50&page=1&sidx=4&sord=desc , if you open it in a new tab, the data is displayed. But if you open that link in other web browser you get 0 results.

That's exactly what is happening to me with NodeJs and Phantom and i don't know how to fix it.

Answer 1

If you want to give Scrapy a try, https://docs.scrapy.org/en/latest/topics/dynamic-content.html explains how to deal with this type of scenarios, and I would suggest reading it after completing the tutorial.

The page can also be handy if you use other scraping framework, as there's not much that is Scrapy-specific, and for Python-specific stuff I'm sure there will be JavaScript counterparts.

As for Cheerio and Phantom, I'm not familiar with them, but it is most likely doable with them as well.

It's doable with any web client, it's just a matter of knowing how to use the tool for this purpose. Most of the work involves using your web browser tools to understand how the website works underneath.