KeyError in python saying KeyError : 'value'

Question

I am trying to get the hidden elements in twitter login page. I followed a procedure which simply gets the hidden elements in that page. But the problem is when i try to get value of those elements, i am getting key error. the code is:

    import requests, lxml.html
from bs4 import BeautifulSoup
s = requests.session()
login = s.get('https://twitter.com/login')
login_html = lxml.html.fromstring(login.text)
hidden_inputs = login_html.xpath(r'//form//input[@type="hidden"]')
form = {x.attrib["name"]: x.attrib["value"] for x in hidden_inputs}
print(form)

I am getting error at x.attrib['value']. How to rectify this?

Answer 1

Here is an example of (some) the objects you will get:

<InputElement 1a62c5ef778 name='ui_metrics' type='hidden'>

There is no "value" key.

If you print this:

for x in hidden_inputs:
     print(x.attrib)

Then you will be able to see which tags have values:

{'type': 'hidden', 'name': 'authenticity_token', 'value': '7fca6a14828cd425dad8437cc291687fc2ff1f96'}

So you will have to explicitly check for the ones that do have values

Answer 2

I use google devtools check twitter login page and get this image. The last 2 inputs either do not have value or it is not key value pair, so I guess that's why you got the error.

Answer 3

This error indicates that a member x of hidden_inputs is not a dictionary including the key "value". You should print out hidden_inputs to see its elements, and make sure they are dictionaries that include the key "value".