用 Beautiful Soup 解析跨度：'NoneType' object 没有属性 'text'

Question

I'm trying to get all relationship names on a Linkedins's web page (the example: https://www.linkedin.com/in/diversiti/detail/skills/(ACoAACfEjjEBNLPrc1Y8OKosqroRRScfwaCdrxI,5)/ )

(Please note the ')' char before the '5').

Here is a part of the html code:

<div class="pv-endorsement-entity__detail  pl3">
    <div class="pv-endorsement-entity__name t-16 t-black t-bold truncated-text">
        <span class="pv-endorsement-entity__name--has-hover">Vignesh G</span>
        <span data-test-distance-badge="" id="ember122"
            class="distance-badge t-black--light t-14 separator t-black--light ember-view"><span
                class="visually-hidden">
                out of network
            </span>
            <span class="dist-value" aria-hidden="true">3rd+</span>
        </span>
    </div>
    <div class="pv-endorsement-entity__headline t-14 t-black--light t-normal">
        Inventor | Engineer | MBA
    </div>
</div>

I want to get the name, so in this case "Vignesh G".

Here is my python code:

from bs4 import BeautifulSoup
from requests_html import HTMLSession
session = HTMLSession()
response = session.get('https://www.linkedin.com/in/diversiti/detail/skills/(ACoAACfEjjEBNLPrc1Y8OKosqroRRScfwaCdrxI,5)/')
soup = BeautifulSoup(response.content, 'html.parser')

content = soup.find('span', {'class': 'pv-endorsement-entity__name--has-hover'}).text

print(content)

Unfortunately I got this error:

'NoneType' object has no attribute 'text'

I suppose that the span object is empty for BeautifulSoup, but how to get the text in this object?

Answer 1

LinkedIn is loading the content later. The initial content does not contain body tag. You should use selenium to simulate a browser.

https://pypi.org/project/selenium/

That way, you can load the URL and wait for the URL to load content completely. It comes with utility functions such as find_element_by_tagname etc, which will work fine as a replacement for BeautifulSoup approach that you are currently taking.