How to get dynamic elements in this case using Selenium/Python?

Question

I try to get the elements that's loaded dynamically to the page after the page has loaded.

Here is the html excluding the scripts last in the html and some other divs before. The comment-div with type="report" is loaded dynamically to the page. I think the site uses Angular or something.

<div class="sc-fUKshf bNdPM">
   <div data-testid="comment" type="report" class="sc-cTmXAz cwRvCX">
      <div class="sc-bCwfaz sc-QfGIp gDveND hNwnuN">
         <div class="sc-hepHJq gyIWHj">
            <svg width="28" height="27" viewBox="0 0 28 27" xmlns="http://www.w3.org/2000/svg" data-testid="report">
               <g transform="translate(.5)" fill="none" fill-rule="evenodd">
                  <rect fill="#E3DFF2" width="27" height="27" rx="10.916"></rect>
                  <path d="M16.009 10.89h-.602a2.73 2.73 0 00-2.723 2.916c.077 1.119.132 1.821.167 2.107.039.327.169 1.096.389 2.31a2.205 2.205 0 002.452 1.792 2.713 2.713 0 002.341-2.328c.128-.951.215-1.543.259-1.774.037-.194.162-.754.376-1.68a2.73 2.73 0 00-2.66-3.343zm6.038 4.327l-.343-.1a1.662 1.662 0 00-2.129 1.598c.001.322.003.52.007.595.005.106.037.398.093.877a1.36 1.36 0 001.515 1.192c.828-.1 1.53-.655 1.82-1.437l.172-.465a1.73 1.73 0 00-1.135-2.26zM7.453 6.436l-.565.057a3.411 3.411 0 00-3.038 3.842c.345 2.597.606 4.268.785 5.014.19.79.641 2.285 1.353 4.485a3.063 3.063 0 003.13 2.112 2.29 2.29 0 002.114-2.53c-.265-2.443-.409-4.119-.432-5.028-.02-.767.072-2.217.276-4.35a3.305 3.305 0 00-3.623-3.602z" fill="#8F7FCE"></path>
               </g>
            </svg>
            <div data-testid="activity-title" class="sc-dmiYbj bEyWbu"><span class="sc-jVBfSZ keEVDi">Weekly report</span></div>
         </div>
         <div class="sc-bCwfaz sc-jomqko hGiREx gbkTPQ">
            <div class="sc-bXXDC jSxSkF">28 May 2021</div>
         </div>
      </div>
      Some text
      <div data-testid="comment-reply-button" class="sc-clYhRO dZSmwI">
         <svg data-name="Layer 1" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 21 21.97" width="21" height="22" class="sc-fdnhOL dYoRsn">
            <path d="M19.03 8.35v10.81a1.81 1.81 0 01-1.8 1.81H2.81A1.81 1.81 0 011 19.16V4.74a1.8 1.8 0 011.81-1.8h9" fill="none" stroke="#384a5f" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2"></path>
            <path d="M19.64 3.58a1.57 1.57 0 10-2.41-2l-7.21 7.67v2.7l2.7-.9z" fill="none" stroke="#384a5f" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2"></path>
         </svg>
         Text
      </div>
      <div class="sc-iArHnM eEOXNH">
         <form data-testid="reply-form">
            <div class="sc-jfkLlK fHByXR"><textarea data-testid="text-area" placeholder="Text" name="replyMessage" id="replyMessage" rows="3" height="auto" class="sc-fcmMJX ldYVmw"></textarea></div>
            <div class="sc-bHCRaJ gPCKhJ"><button data-testid="comment-reply-cancel-button" type="button" class="sc-ckTSus sc-fzJAIQ kiNKGC irzfTA">Text</button><button data-testid="comment-reply-submit-button" type="submit" class="sc-lbVvki hxgvIg">Text</button></div>
         </form>
      </div>
   </div>
 </div>

Here is the python/selenium code:

relativeXpathToReportComments = "//div[@type='report']"
    
try: 
    time.sleep(5) #explicit wait in seconds. 
    element = WebDriverWait(driver, 90).until(EC.element_to_be_clickable((By.XPATH, relativeXpathToReportComments))) #does not always work?

    comments = driver.find_elements_by_xpath(relativeXpathToReportComments) #list all elements with comments with type Report
    print("You found the elements for comments:" + comments)
    length = len(comments)
    print (length)
    commentReport = comments[0].get_attribute("innerText").split('\n')[2] #get all the strings inside the element from the first/latest report, in a list with split, then extract only the right comment string (not the title, date etc)
    print("Your last report was:" + commentReport)
except:
    print("could not find/get comments on comment page")

90 % of the time I get the result "Could not find/get comments on comment page". It never works on the first run of the code, but when I call a function again that executes the code above it works 5-10 % of the times.

Answer 1

I guess the "//div[@type='report']" locator is not unique, so some other element is caught by Selenium with the expected condition while that element is not clickable.