Scrape web with login session (using python scrapy)

Question

I am trying to extra some data under a login session using y. Usually when I intercept data I can view the format data in the network tab. In this login page, when the log on is submitted, I can't seems to find the format data tab.

https://lumen.ni.com/nicif/login.xhtml

Can anyone advice?

My current implementation:

import y
from y import FormReuest

class LoginSpider(y.Spider):
name = 'example.com'


def start_requests(self):
    return [y.FormRequest("https://lumen.ni.com/nicif/login.xhtml",
                               formdata={'LoginForm:email': 'email', 'LoginForm:password': 'password'},
                               callback=self.logged_in)]

def logged_in(self, response):
    if("Please use correct email & password combination") in str(response.body):
        print("fail")
    else:
        data = {'p_part_number':'779494-01','p_duration':'','p_duration_uom':''}
        yield FormRequest(url="http://sine.ni.com/apps/utf8/nipp.purchase_form_ajax", formdata=data)

Answer 1

There are lots of other parameters they are using along with form, they are hidden in the login form, you need to send them too along with your request.