Selenium xpath 无法定位 class

Question

I am working on a project to scrape data. I have a for loop that runs through 50 urls (all of which are the same page with just different information) and then I extract different things to add to a csv. The problem I am having is that when I try to extract 'job_title' in my code, many of the entries come up as 'None', though the entry is actually existent. The HTML seems to be same in each URL, but 10/50 urls are yielding 'NONE' to the following lines of code. I need the code to set job_title = 'Founder'

This is the code I am currently using:

sel = Selector(text=driver.page_source) 
job_title = sel.xpath('//*[starts-with(@class, "t-16 t-black t-bold")]/text()').extract_first()

Here is the HTML from one of the urls that I was unable to extract job_title--Which is 'Founder' in this case. It is the second line of the script.

 <div class="pv-entity__summary-info pv-entity__summary-info--background-section mb2"> <h3 class="t-16 t-black t-bold">Founder</h3> <p class="visually-hidden">Company Name</p> <p class="pv-entity__secondary-title t-14 t-black t-normal"> Genamint <span class="pv-entity__secondary-title separator">Full-time</span> </p> <div class="display-flex"> <h4 class="pv-entity__date-range t-14 t-black--light t-normal"> <span class="visually-hidden">Dates Employed</span> <span>Mar 2020 – Present</span> </h4> <h4 class="t-14 t-black--light t-normal"> <span class="visually-hidden">Employment Duration</span> <span class="pv-entity__bullet-item-v2">5 mos</span> </h4> </div> <h4 class="pv-entity__location t-14 t-black--light t-normal block"> <span class="visually-hidden">Location</span> <span>New York, United States</span> </h4> <!----> </div>

Any help would be appreciated.

Answer 1

Both those lines grab this HTML. <h3 class="nav-settings__member-name t-16 t-black t-bold> Ethan Roberti </h3>

There's no nav-settings__member-name in your sample data. Since you're using extract_first() , you get the first appearing result. One way to fix it would be:

(//div[contains(@class,"pv-entity__summary")])[1]//h3/text()

Output: Founder

Assuming you're trying to scrape LinkedIn, to get the current or last job of a person, use the following XPath:

(//section[@class="experience pp-section" or @id="experience-section"]//h3)[1]/text()

For example, for https://www.linkedin.com/in/ethan-roberti-322694174 , you'll get:

Output: Summer Analyst