使用beautifulSoup在元素中抓取数据

Question

I am doing web scraping using beautifulSoup. I managed to scrape the name, however the problem is, I don't really sure how to scrape if the data is in the elements for example the phone number and email are located as in the pictures below:

My code:

 import requests
    from bs4 import BeautifulSoup

    raw = requests.get('https://www.iproperty.com.my/property/findanagent.aspx?ty=as&ak=&rk=&pg=1&rmp=10&st=KL&ct=&st1=&ct1=#40091').text
    raw = raw.replace("</br>", "")

    soup = BeautifulSoup(raw, 'html.parser')

    import re

phone = ['data-content'])[0][1:][:-1] for d in soup.find_all('a',{'class':'csagentphonelead'})]
    name = [x.text.strip().split("\r\n")[-1].strip() for x in soup.find_all("p", class_='box-listing_agentCS')]
    website = [x.text.strip().split("\r\n")[-1].strip() for x in soup.find_all("a", class_='csagentemaillead')] 

    num_page_items = len(name)
    with open('results180.csv', 'a') as f:
        for i in range(num_page_items):
        f.write(name[i] + "," + phone[i] + "," + website[i] + "," + "\n")

My results from scraping is "Click for Email" and "Click for Phone". What should I fix so that the results are proper email and phone number?

Answer 1

You have to get the value of data attribute from the link. You can try this code -

import requests
from bs4 import BeautifulSoup

raw = requests.get('https://www.iproperty.com.my/property/findanagent.aspx?ty=as&ak=&rk=&pg=1&rmp=10&st=KL&ct=&st1=&ct1=#40091').text
raw = raw.replace("</br>", "")

soup = BeautifulSoup(raw, 'html.parser')

import re
#['data-content'])[0][1:][:-1] ## note sure what is this
# for d in soup.find_all('a',{'class':'csagentphonelead'}):
name = [x.text.strip().split("\r\n")[-1].strip() for x in soup.find_all("p", class_='box-listing_agentCS')]
phone = [x['data'].strip().split("\r\n")[-1].strip() for x in soup.find_all("a", class_='csagentphonelead')] 
website = [x['data'].strip().split("\r\n")[-1].strip() for x in soup.find_all("a", class_='csagentemaillead')] 

num_page_items = len(name)
with open('results180.csv', 'a') as f:
    for i in range(num_page_items):
        f.write(name[i] + "," + phone[i] + "," + website[i] + "," + "\n")