How to scrape drop-down menu options using Selenium Webdriver with Python?

Question

I use Selenium Webdriver to test a web site with a drop-down menu with different options for different users. The number of options and their values are always different. When I look at the source, I see the code below. Could you please provide an example of how in Python I can scrape it and make a list of all the available option values?

<div _ngcontent-pxo-26="" class="col-md-6">
  <div _ngcontent-pxo-26="" class="form-group">
    <label _ngcontent-pxo-26="" for="Filter_ClientRegion">Region</label>
    <select _ngcontent-pxo-26="" class="form-control ng-pristine ng-valid ng-touched" id="Filter_ClientRegion">
      <option _ngcontent-pxo-26="" value="">All</option>
      <!--template bindings={}--
      <option _ngcontent-pxo-26="" value="A">A</option>
      <option _ngcontent-pxo-26="" value="B">B</option>
      <option _ngcontent-pxo-26="" value="C">C</option>
      <option _ngcontent-pxo-26="" value="D">D</option>
      <option _ngcontent-pxo-26="" value="E">E</option>
      <option _ngcontent-pxo-26="" value="F">F</option>
      <option _ngcontent-pxo-26="" value="G">G</option>
    </select>
  </div>
</div>

Answer 1

To select a specific option , you can use something like:

from selenium import webdriver
driver = webdriver.Firefox()

driver.get("some.site")
el = driver.find_element_by_id('Filter_ClientRegion')
for option in el.find_elements_by_tag_name('option'):
    if option.text == 'A': # or  B or C...
        option.click() # select() for older versions
        break

---

To get the values of option , you can use:

options = []
driver.get("some.site")
el = driver.find_element_by_id('Filter_ClientRegion')
for option in el.find_elements_by_tag_name('option'):
    options.append(option.get_attribute("value"))
# print(options)
# A B C ...

---

Notes:
1. I cannot fully test the code above because I don't have the complete source code
2. Please note that the options code is inside a comments block <!--template bindings={}-- and you may not be able to retrieve its value.

Answer 2

It should be pretty easy.

array_options =  []
element = WebDriverWait(self.driver, timeout=wait_time).until(
          EC.visibility_of_element_located("id","Filter_ClientRegion")))
if element.tag_name == 'select':
    select = Select(element)
    dropdown_options = select.options
    for option in dropdown_options:
        array_options.append(option.text)

Answer 3

You can do this with BeautifulSoup.

Since you mentioned selenium this code begins by using that, in case you need it to get past a login or something else requiring selenium. If you don't need selenium then you can skip down to the line where soup is made using BeautifulSoup . The preceding code just shows how to use selenium to get source code so that it can be accessed by BeautifulSoup .

First find the select tag that contains all of the HTML code, including the commented stuff. Then take each item in this list, convert it to a string and concatenate it into one big string, and prepend <select> . Turn this big string into soup and findAll the option tags within it. Display whatever stuff you want from each of these tags.

>>> from selenium import webdriver
>>> driver = webdriver.Chrome()
>>> content = driver.page_source
>>> from bs4 import BeautifulSoup
>>> soup = BeautifulSoup(content, 'lxml')
>>> select = soup.find('select', attrs={'id': 'Filter_ClientRegion'})
>>> items = []
>>> for item in select.contents:
...     items.append(str(item).strip())
...     
>>> items
['', '<option _ngcontent-pxo-26="" value="">All</option>', '', 'template bindings={}--\n      <option _ngcontent-pxo-26="" value="A">A</option>\n      <option _ngcontent-pxo-26="" value="B">B</option>\n      <option _ngcontent-pxo-26="" value="C">C</option>\n      <option _ngcontent-pxo-26="" value="D">D</option>\n      <option _ngcontent-pxo-26="" value="E">E</option>\n      <option _ngcontent-pxo-26="" value="F">F</option>\n      <option _ngcontent-pxo-26="" value="G">G</option>\n    </select>\n  </div>\n</div>']
>>> newContents = '<select>' + ''.join(items).replace('--','')
>>> newSelectSoup = BeautifulSoup(newContents)
>>> options = newSelectSoup.findAll('option')
>>> len(options)
8
>>> for option in options:
...     option.attrs['value']
...     
''
'A'
'B'
'C'
'D'
'E'
'F'
'G'