Alternative to time.sleep() in selenium using python while web scraping?

Question

I need to scrape price of certain listed food items basis different locations in the country. There's an input text box that allows me to enter the name of the city & pressing "Enter" shows me the list of items available in that city.

Here's how I am trying to automate this:

driver.get("https://grofers.com/")
ele = driver.find_element_by_xpath("//input[@data-test-id='area-input-box']")`
ele.send_keys(area)
ele.send_keys(Keys.RETURN)

Here's the HTML I'm working with:

 <div style="margin-left: 51px; height: 36px;"> <div style="display: flex; height: 100%;"> <button class="btn location-box mask-button">Detect my location</button> <div class="oval-container"> <div class="oval"> <span class="separator-text"> <div class="or">OR</div> </span> </div> </div> <div style="width: 220px;"> <div class="modal-right__input-wrapper"> <div class="display--table full-width"> <div class="display--table-cell full-width"> <div id="map-canvas"></div> <div class="Select location-search-input-v1 is-searchable Select--single"> <div class="Select-control"> <div class="Select-multi-value-wrapper" id="react-select-2--value"> <div class="Select-placeholder">Type your city Society/Colony/Area</div> <div class="Select-input" style="display: inline-block;">**<input data-test-id="area-input-box" aria-activedescendant="react-select-2--value" aria-expanded="false" aria-haspopup="false" aria-owns="" role="combobox" value="">**</div> </div> <span class="Select-arrow-zone"><span class="Select-arrow"></span></span> </div> </div> </div> </div> </div> </div> </div>

The problem is - after send_keys, the website takes time to autofill the input box AFTER WHICH I need to press enter.

I tried using time.sleep(2) after send_keys but this leads to pop-up disappearing & a StaleElementException when I do Keys.RETURN.

Have been stuck on this for quite some time now. Any help/pointers would be appreciated.

Answer 1

I need to scrape price of certain listed food items basis different locations in the country. There's an input text box that allows me to enter the name of the city & pressing "Enter" shows me the list of items available in that city.

Here's how I am trying to automate this:

driver.get("https://grofers.com/")
ele = driver.find_element_by_xpath("//input[@data-test-id='area-input-box']")`
ele.send_keys(area)
ele.send_keys(Keys.RETURN)

Here's the HTML I'm working with:

 <div style="margin-left: 51px; height: 36px;"> <div style="display: flex; height: 100%;"> <button class="btn location-box mask-button">Detect my location</button> <div class="oval-container"> <div class="oval"> <span class="separator-text"> <div class="or">OR</div> </span> </div> </div> <div style="width: 220px;"> <div class="modal-right__input-wrapper"> <div class="display--table full-width"> <div class="display--table-cell full-width"> <div id="map-canvas"></div> <div class="Select location-search-input-v1 is-searchable Select--single"> <div class="Select-control"> <div class="Select-multi-value-wrapper" id="react-select-2--value"> <div class="Select-placeholder">Type your city Society/Colony/Area</div> <div class="Select-input" style="display: inline-block;">**<input data-test-id="area-input-box" aria-activedescendant="react-select-2--value" aria-expanded="false" aria-haspopup="false" aria-owns="" role="combobox" value="">**</div> </div> <span class="Select-arrow-zone"><span class="Select-arrow"></span></span> </div> </div> </div> </div> </div> </div> </div>

The problem is - after send_keys, the website takes time to autofill the input box AFTER WHICH I need to press enter.

I tried using time.sleep(2) after send_keys but this leads to pop-up disappearing & a StaleElementException when I do Keys.RETURN.

Have been stuck on this for quite some time now. Any help/pointers would be appreciated.

Answer 2

Selenium actually has an article on this with Explicit and Implicit waits, I think this is the one you're looking for:

# Wait until an element with id='myNewInput' has class 'myCSSClass'
wait = WebDriverWait(driver, 10)
element = wait.until(element_has_css_class((By.ID, 'myNewInput'), "myCSSClass"))

https://selenium-python.readthedocs.io/waits.html That's the article

Answer 3

You can also create custom wait conditions when none of the previous convenience methods fit your requirements. A custom wait condition can be created using a class with call method which returns False when the condition doesn't match.

 class element_has_css_class(object): """An expectation for checking that an element has a particular css class. locator - used to find the element returns the WebElement once it has the particular css class """ def __init__(self, locator, css_class): self.locator = locator self.css_class = css_class def __call__(self, driver): element = driver.find_element(*self.locator) # Finding the referenced element if self.css_class in element.get_attribute("class"): return element else: return False # Wait until an element with id='myNewInput' has class 'myCSSClass' wait = WebDriverWait(driver, 10) element = wait.until(element_has_css_class((By.ID, 'myNewInput'), "myCSSClass"))