Selenium / Python使用Selenium扩展文本后如何获取全文？

Question

I am trying to scrape reviews from TripAdvisor which, for long reviews, display only partial reviews that require clicking 'More' for the full review to be displayed. I tried getting the text after clicking more (and I can see that the text is expanded) but all I get is the partial review.

My code (to scrape one specific review) is as follow:

driver = webdriver.Firefox()
driver.get(url)
review = driver.find_element_by_id("review_541350982") 
review.find_element_by_class_name("taLnk.ulBlueLinks").click()
driver.wait = WebDriverWait(driver, 5)
new_review = driver.find_element_by_id("review_541350982")
entry = new_review.find_element_by_class_name("partial_entry")
print entry.text

This is the HTML before clicking on 'More':

<p class="partial_entry">This place blah blah blah What an...
<span class="taLnk ulBlueLinks" onclick="widgetEvCall('handlers.clickExpand',event,this);">More</span>
</p>

and this is the HTML after:

<p class="partial_entry">This place blah blah blah What an incredible monument from both a historic and construction point of view.</p>
<span class="taLnk ulBlueLinks" onclick="widgetEvCall('handlers.clickCollapse',event,this);">Show less</span>

I noticed that now <span> comes after <p> after clicking 'More'. Not sure if this is useful.

Any advice is greatly appreciated!

EDIT: Noticed that introducing time.sleep(1) instead of driver.wait solved the problem. Wonder if there is a better way to do this such that the new entry is obtained automatically after it changes and not having to set an arbitrary waiting time?

Answer 1

It is pretty much evident from your code that the WebDriverWait though defined but was not used properly. To print the full text This place blah blah blah What an incredible monument from both a historic and construction point of view. , you can use the following code block :

from selenium.webdriver.support import expected_conditions as EC
#code block
review = driver.find_element_by_id("review_541350982") 
review.find_element_by_class_name("taLnk.ulBlueLinks").click()
new_review = driver.find_element_by_id("review_541350982")
full_review = WebDriverWait(driver, 10).until(EC.text_to_be_present_in_element(new_review.find_element_by_xpath("//p[@class='partial_entry']"),'This place blah blah blah What an incredible monument from both a historic and construction point of view.'))
entry = new_review.find_element_by_class_name("partial_entry")
print entry.text

Answer 2

Locate the review and click more:

review = driver.find_element_by_id("review_541350982")
partial_text = review.find_element_by_tag_name('p')
partial_text.find_element_by_tag_name('span').click()

Relocate the review using XPath and output the text:

new_review = driver.find_element_by_xpath('(//*[@id="review_541350982"]//p)[1]')
print(new_review.text)

HTH