使用漂亮的 Soup 从 Airtasker 中提取数据

Question

I am trying to extract data from this website - https://www.airtasker.com/users/brad-n-11346775/ .

So far, I have managed to extract everything except the license number . The problem I'm facing is bizarre as the license number is in the form of text. I was able to extract everything else like the Name, Address etc. For example, to extract the Name, I just did this:

name.append(pro.find('div', class_= 'name').text)

And it works just fine.

This is what I have tried to do, but I'm getting the output as None

license_number.append(pro.find('div', class_= 'sub-text'))

When I do:

license_number.append(pro.find('div', class_= 'sub-text').text) 

It gives me the following error:

AttributeError: 'NoneType' object has no attribute 'text'

That means it does not recognise the license number as a text, even though it is a text.

Can someone please give me a workable solution and please tell me what am I doing wrong??? Regards,

Answer 1

The badge with the license number is added to the HTML dynamically from a Boostrap JSON that sits in one of the <script> tags.

You can find the tag with bs4 and scoop out the data with regex and parse it with json .

Here's how:

import ast
import json
import re

import requests
from bs4 import BeautifulSoup

page = requests.get("https://www.airtasker.com/users/brad-n-11346775/").text
scripts = BeautifulSoup(page, "lxml").find_all("script")[-4]
bootstrap_JSON = json.loads(
    ast.literal_eval(re.search(r"parse\((.*)\)", scripts.string).group(1))
)
print(bootstrap_JSON["profile"]["badges"]["electrical_vic"]["reference_code"])

Output:

Licence No. 28661