scrape the value from a dropdown

Question

I am trying to web scrape the value and text from a dropdown element on a webpage using a combination of Python with selenium and Beautiful Soup.

I am able to get the text but I am not able to get the value through the get_attribute command.

When I print the element that I located on the webpage it returns the following content

The print statement that gets it gives the error:

None Type object is not callable

price=soup.find("select",{"id":"space-prices"})
print(price)
print(price.text)
print(price.get_attribute('value'))

The output for print(price) is

<select class="pricing-bar-select" id="space-prices" name="space-prices"><option selected="selected" value="£360">Per Day</option>
<option value="£1,260">Per Week</option>
<option value="£5,460">Per Month</option>
<option value="£16,380">Per Quarter</option>
<option value="£65,520">Per Year</option></select>

The URL of the webpage is

https://www.appearhere.co.uk/spaces/north-kensington-upcycling-store-and-cafe

Answer 1

try this:

from selenium import webdriver
from bs4 import BeautifulSoup


driver = webdriver.Chrome()
url= "https://www.appearhere.co.uk/spaces/north-kensington-upcycling-store-and-cafe"
driver.maximize_window()
driver.get(url)

content = driver.page_source.encode('utf-8').strip()
soup = BeautifulSoup(content,"html.parser")
price=soup.find("select",{"id":"space-prices"})
options = price.find_all("option")
options1=[y.text for y in options]
values = [o.get("value") for o in options]
for x in range(5):
    print options1[x], values[x].encode('utf8')
driver.quit()

It will print

Per Day £360
Per Week £1,260
Per Month £5,460
Per Quarter £16,380
Per Year £65,520

Hope this is what you want

Answer 2

It's because get_attribute seems to be None . It's not a valid attribute of the prices object. So it's not a function that you can call - hence the error. If you took away the parentheses and just printed prices.get_attribute nothing would print, because the value is None .

Also, the <select> tag doesn't have a "value" attribute in the first place. What you've done is you've grabbed the <select> tag, and all of it's children. Each child in the <select> tag (the <option> tags) have a "value" attribute. If you are trying to get all of the values of all of the <option> tags in that <select> , then you should do the following:

price=soup.find("select",{"id":"space-prices"})

# get all <options> in a list
options = price.find_all("option")

# for each element in that list, pull out the "value" attribute
values = [o.get("value") for o in options]
print(values)
#[u'\xa3360', u'\xa31,260', u'\xa35,460', u'\xa316,380', u'\xa365,520']