使用BeautifulSoup从网页检索链接

Question

I am trying to pull links from a webpage at a certain position, then open that link, and then repeat that process for the provided number of times. The problem is I keep getting the same URL returned, so it seems like my code is just pulling the tag, printing the tag, not opening it, and doing that process X number of times before closing.

I have written and re-written this code a number of times, but for the life of me I just can't figure it out. Please tell me what I am doing wrong

Tried using list to put anchor tags in, then open the url at the requested position in the list, then do a list clear before starting the loop over again.

import urllib.request, urllib.parse, urllib.error
from bs4 import BeautifulSoup
import ssl

# Ignore SSL certificate errors
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

#url = input('Enter - ')
url = "http://py4e-data.dr-chuck.net/known_by_Fikret.html"
html = urllib.request.urlopen(url, context=ctx).read()
soup = BeautifulSoup(html, 'html.parser')

count = 0 
url_loop = int(input("Enter how many times to loop through: ")) 
url_pos= int(input("Enter position of URL: "))
url_pos = url_pos - 1

print(url_pos)



# Retrieve all of the anchor tags
tags = soup('a')
while True:
    if url_loop == count:
        break
    html = urllib.request.urlopen(url, context=ctx).read()
    soup = BeautifulSoup(html, 'html.parser')
    url = tags[url_pos].get('href', None)

    print("Acquiring URL: ", url)

    count = count + 1  

print("final URL:", url)

Answer 1

it could be that the tags are only extracted once for the initial document:

# Retrieve all of the anchor tags
tags = soup('a')

If you were to re-extract the tags after fetching each document, they would reflect the last document.