在使用 python selenium 单击查看联系人后，我无法获取相同的检查 output

Question

from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver import ActionChains
from selenium.common.exceptions import TimeoutException

driver = webdriver.Chrome(r'C:\chromedriver.exe')

driver.get('https://www.gigadocs.com/hyderabad/dentist')

driver.find_element_by_xpath('//*[@id="listingTab"]/div[2]/div/div[1]/div[1]/div/div[2]/div[2]/ul/li[1]/span'). click()

soup = BeautifulSoup(driver.page_source,'html.parser')

mobile = soup.find('ul',class_='detailsList')
print(mobile)

i am trying to click on view contact to scrape the mobile number but after clicking, getting same output as view contact.

Answer 1

Induce WebDriverWait and element_to_be_clickable() and click the view contact and then get the li tag under ul tag.

from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
driver = webdriver.Chrome(r'C:\chromedriver.exe')
driver.get('https://www.gigadocs.com/hyderabad/dentist')
WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.XPATH,"//span[@data-source='mobile'][text()='View Contact']"))).click()
time.sleep(1)
soup = BeautifulSoup(driver.page_source,'html.parser')
mobile = soup.find('ul',class_='detailsList')
print(mobile.find('li').text)

Answer 2

You don't need the overhead of selenium. The page makes POST requests using ids for doctor and clinic to retrieve telephone numbers. You can scrape these ids from the initial page then mimic those requests to get the tel numbers. I use doctor id as the key for a dictionary and update the values with the tel number.

import requests
from bs4 import BeautifulSoup as bs

headers = {
    'User-Agent': 'Mozilla/5.0',
    'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
    'X-Requested-With': 'XMLHttpRequest'
}

data = {
  'doctorId': '3806',   #[data-doctor]
  'clinicId': '1519',   #[data-clinic]
  'clickSource': 'mobile'
}

with requests.Session() as s:
    s.headers = headers
    r = s.get('https://www.gigadocs.com/hyderabad/dentist')
    soup = bs(r.content, 'lxml')
    tel_numbers = {i['data-doctor']:i['data-clinic'] for i in soup.select('.appointmentBtn')}

    for k, v in tel_numbers.items():
        data['doctorId'] = k
        data['clinicId'] = v
        r = s.post('https://www.gigadocs.com/search/getmobilenumbers', data=data).json()
        tel_numbers[k] = r['mobile']

print(tel_numbers)