Python Beautiful Soup 我想进入标签元素内部

Question

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin
import csv

headers = {"Accept-Language": "es-ES,es;q=0.9",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36"}
url = 'https://mydramalist.com/search?adv=people&na=3&so=popular&page=1'

above link i want to scrape from..........................................................................................................................................................

while True:
        print(url)
        response = requests.get(url, headers=headers)
        # print(response.status_code)
        soup = BeautifulSoup(response.content, 'html.parser')
        footer = soup.select_one('li.page-item.nb.active')
        print(footer.text.strip())
        for tags in soup.find_all('h6'):
            print(tags)
            # tags = soup.select_one('h6>a') <<<<<<<<<<< This part i want to go inside of h6 element click it and get data from there
        next_page = soup.select_one('li.page-item.next>a')
        if next_page:
            next_url = next_page.get('href')
            url = urljoin(url, next_url)
        else:
            break


Hi Guys, I want to extract data from current page, going to clickable page which is the h6 tag. and loop again. I cannot figure out how can I solve the issue with for loops. please help thank you. i already updated the code

Answer 1

From the url you provided, taking the first as an example,

Notice there /people/232-lee-min-ho is a sublink.

All you got to do is scrape the sublink and add it to the main link as shown below,

new_link = https://mydramalist.com + sublink

it should give you the full link https://mydramalist.com/people/232-lee-min-ho

Now perform another requests.get(new_link) on your new link to retrieve the contents.