How to crawl a website that requires login using scrapy?

Question

I want to scrape data from a website, which requires a login to reach a certain page only then the data can be scraped.

Is there any way to scrape the data after login using Scrapy ? or if we can simulate the login ?

Note: I do have the login credentials with me.

Answer 1

Short answer : Yes, you can scrape data after login. Check Formdata in scrapy and this answer post request using scrapy and documentation

Long Answer : Login pages are just forms. You can access those fields to fill in the required details and post that data. You can manually login and check the chrome developer tools [ctrl + shift + i] for network call being made when you press the submit/login button. You can then inspect the post request made and duplicate it in your scraper. You can check the above links to read about how to post data, and how requests and responses work in scrapy.