使用美丽的汤从网页中的链接中抓取数据。 Python

Question

I am trying to scrape data (instaid, average likes, average comments) from a url inside the webpage: , https://starngage.com/app/global/influencer/ranking/india

The element id of the url is : @priyankachopra

Similary I want to scrape data from all 1000 profiles in the same table

Can someone tell me how to do this

from bs4 import BeautifulSoup
from prettytable import PrettyTable

tb = PrettyTable(['Name', 'Insta_ID', 'Followers'])
url = 'https://starngage.com/app/global/influencer/ranking/india'
resp = requests.get(url)

soup = BeautifulSoup(resp.text, 'html.parser')

table = soup.find('table', class_='table-responsive-sm')
td = table.findAll('tr')

for i in td[1:]:
    temp = i.select_one("td:nth-of-type(3)").text
    name, insta_id = temp.split('@')
    followers = i.select_one("td:nth-of-type(6)").text
    tb.add_row([name.strip(), insta_id.strip(), followers.strip()])

print(tb)

Answer 1

You can do this, I hadn't exactly tested complete code because it will take very much time it may take upto 10mins but I had tested part part and is working perfectly fine for me. But if not working ask me in comment. Here's code:

import requests
from bs4 import BeautifulSoup
import pandas as pd
import re

ids=[]
avgc=[]
avgl=[]
for i in range(1,101):
    url = f'https://starngage.com/app/global/influencer/ranking/india?page={i}'
    print(url)
    resp = requests.get(url)
    
    soup = BeautifulSoup(resp.text, 'lxml')
    
    table = soup.find('table', class_='table-responsive-sm')
    trs = table.findAll('tr')
    
    for tr in trs[1:]:
        temp = tr.select_one("td:nth-of-type(3)").text
        _,insta_id = temp.split('@')
        ids.append(insta_id.strip())

for id in ids:
    page=requests.get("https://starngage.com/app/global/influencers/"+id)
    soup=BeautifulSoup(page.content, 'lxml')
    
    x=soup.find("blockquote").find("p").text.strip()
    #You can change this re code. I am not much familar with re. So, if you find any other better approch then comment.
    x=re.findall("is \d+",x)
    avl,avc=list(map(lambda y: y.replace("is ",""),x))
    avgl.append(avl)
    avgc.append(avc)

df = pd.DataFrame({"Insta Id":ids,"Avgerage Like":avgl,"Avgerage Commment":avgc})

print(df)

df.to_csv("test.csv")