Beautiful Soup find is returning none

Question

found is returning 'none'

Here is all of the code I've tried and the html I am working with:

url = "https://www.instagram.com/p/BszEBehhwet/"
a = urlopen(url)
html = a.read()
a.close()
page_soup = soup(html, "html.parser")

found = page_soup.find("div", {"class":"P9YgZ"})

<div class="KlCQn G14m- EtaWk">
    <ul class="k59kT">
        <li class="gElp9 " role="menuitem">
            <div class="P9YgZ">
                <div class="C7I1f X7jCj">
                    <div class="C4VMK">
                        <h2 class="_6lAjh">
                            <a class="FPmhX notranslate TlrDj" 
                            title="ray.walker00" 
                            href="/ray.walker00/">ray.walker00
                            </a>
                        </h2>
                        <span>Jan. 18, 2019 // Awesome
</span>
</div>
</div>
</div>
</li>
</ul>
</div>

I'd like to return the div class P9YgZ

Answer 1

As I stated in the comments, the page you're working with relies so heavily on javascript that urllib alone will not cut it. Here's an example utilizing Selenium WebDriver that gets the element with that class. You will need to download ChromeDriver and modify the code to point it to where it's located on your system:

from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.options import Options


def main():

    options = Options()
    options.add_argument("--headless")

    driver = webdriver.Chrome(
        options=options, executable_path="C:\chromedriver\chromedriver.exe"
    )

    try:
        driver.get("https://www.instagram.com/p/BszEBehhwet/")

        soup = BeautifulSoup(driver.page_source, "html.parser")
        print(soup.find("div", {"class": "P9YgZ"}))

    finally:
        driver.quit()


if __name__ == "__main__":
    main()

Result:

<div class="P9YgZ"><div class="C7I1f X7jCj"><div class="C4VMK"><h2 class="_6lAjh"><a class="FPmhX notranslate TlrDj" href="/thetremason/" title="thetremason">thetremason</a></h2><span>How I’m finna pull up to ya function.</span></div></div></div>