Can Selenium WebDriver click on all combinations from drop down menu

Question

So basically I need to get all combinations of Department, Course, and Section from the UCLA bookstore site: http://shop.uclastore.com/courselistbuilder.aspx and then I need to choose the books and parse the resulting html page.

Rather than doing it manually (which would take forever), I was looking up other options to do it programmatically. One option I found is Selenium WebDriver which deals with browser automation. Looking at examples from SO I find that the Selenium WebDriver is a promising feature but I am not quite sure if it can be able to do what I need it to.

More or less in pseudocode format, here is my approach on how I am going to use selenium web driver

go to the site: http://shop.uclastore.com/courselistbuilder.aspx

for each_department in department:
    click on each_department
    for each_course in course:
        click on each_course
            for each_section in section:
                click on each_section
// After every department, course, and section has been chosen, we click choose books
click on choose books link

// Save the resulting html file
save next page as html file

I wanted to know if I can be able to do what I want using Selenium WebDriver. It would be helpful if someone could offer better pseudocode that would be tied more appropriately with Selenium WebDriver but I am mainly focused on if this functionality could be possible. I also wanted to mention that I am planning to use the Python API when using Selenium.

Answer 1

This is there you should start (explanation below):

from selenium import webdriver
import time
from selenium.webdriver.support.select import Select


url = "http://shop.uclastore.com/courselistbuilder.aspx"

driver = webdriver.Firefox()
driver.get(url)
time.sleep(1)

departments = Select(driver.find_element_by_id('clDeptSelectBox'))
for department in departments.options:
    # select department
    departments.select_by_value(department.get_attribute('value'))
    time.sleep(1)

    cources = Select(driver.find_element_by_id('clCourseSelectBox'))
    for cource in cources.options:
        # select course
        cources.select_by_value(cource.get_attribute('value'))
        time.sleep(1)

        sections = Select(driver.find_element_by_id('clSectionSelectBox'))
        for section in sections.options:
            print {'department': department.text,
                   'course': cource.text,
                   'section': section.text}

driver.close()

Prints:

{'department': u'AFRCST - AFRICAN STUDIES', 'course': u'201A', 'section': u'1 - LYDON'}
{'department': u'AFRKLA - AFRICAN LANGUAGES', 'course': u'1A', 'section': u'1 - TA'}
{'department': u'AFRKLA - AFRICAN LANGUAGES', 'course': u'150A', 'section': u'1 - FANTA, A.A.'}
{'department': u'AFROAM - AFRO-AMERICAN STUDIES', 'course': u'6', 'section': u'1 - STREETER, C.A.'}
{'department': u'AFROAM - AFRO-AMERICAN STUDIES', 'course': u'M10A', 'section': u'1 - LYDON, G.E.'}
{'department': u'AFROAM - AFRO-AMERICAN STUDIES', 'course': u'M102', 'section': u'1 - LEWIS, L.I.'}
{'department': u'AFROAM - AFRO-AMERICAN STUDIES', 'course': u'M103A', 'section': u'1 - PRICE, Z.F.'}
{'department': u'AFROAM - AFRO-AMERICAN STUDIES', 'course': u'M104A', 'section': u'1 - YARBOROUGH'}
...

The idea is to extensively use Select class that provides a nice API the select/options functionality. First, we are getting all departments, then iterating over the options and choose the next department in the loop. Then, after the small delay, we are getting the list of courses and sections in the same manner.

I've left for you a better handling of Waits ( time.sleep() really is not very reliable) and clicking on Choose Books button (well, German Petrov provided you with both).

Hope that helps.

Answer 2

You can click all options using ~ such code:

select_dept = driver.find_element_by_id('clDeptSelectBox')
for option_dept in select_dept.find_elements_by_tag_name('option'):
    option_dept.click()
    #wait until course has options
    wait.until(lambda driver: driver.find_element_by_xpath("//select[@id='clCourseSelectBox']//option"))
    select_course = driver.find_element_by_id('clCourseSelectBox')
    for option_course in select_course.find_elements_by_tag_name('option'):
         option_course.click()
         #wait until section has options
         ....

and the same for section select, then waiting for link to present and click:

wait.until(lambda driver: driver.find_element_by_xpath("//a[contains(., 'Choose Books')]"))
driver.find_element_by_xpath("//a[contains(., 'Choose Books')]").click()