递归查找所有链接使用bs4和Python问题

Question

I'm using the below code to recursively collect all links from the given website the only problem is that I get this in the beginning of the output file:

https://www.metatrader4.com/en/trading-platform/help/beginning/autotrading/en

https://www.metatrader4.com/en/trading-platform/help/beginning/autotrading/en/en

https://www.metatrader4.com/en/trading-platform/help/beginning/autotrading/en/en/en

...

https://www.metatrader4.com/en/trading-platform/help/beginning/autotrading/en/en/en/en/en/en/en/en/en/en/en/en

etc..

How can I prevent/remove this?

The code:

from bs4 import BeautifulSoup
import requests

# lists 
urls=[] 
   
# function created 
def scrape(site): 
       
    # getting the request from url 
    r = requests.get(site) 
       
    # converting the text 
    s = BeautifulSoup(r.text,"html.parser") 
       
    for i in s.find_all("a"):
          
        href = i.attrs['href'] 
           
        if href.startswith("/"): 
            site = site+href
               
            if site not in  urls: 
                urls.append(site)  
                print(site) 
                # calling it self 
                scrape(site) 
   
# main function 
if __name__ =="__main__": 
   
    # website to be scrape 
    site="https://www.metatrader4.com/en/trading-platform/help/beginning/autotrading"
   
    # calling function 
    scrape(site)

Answer 1

What happens?

Your are replacing the site with itself + the href and first href it finds only contains /en .

How to fix?

Instead of site use a baseUrl

baseUrl = "https://www.metatrader4.com/"

Not sure why you are calling scrape() over and over again I commented it out in my example, cause it is not necessary.

Example

from bs4 import BeautifulSoup
import requests

# lists 
urls=[] 
   
# function created 
def scrape(site, baseUrl): 
       
    # getting the request from url 
    r = requests.get(site) 
       
    # converting the text 
    s = BeautifulSoup(r.text,"html.parser") 
       
    for i in s.find_all("a"):
          
        href = i.attrs['href'] 
           
        if href.startswith("/"): 
            site = baseUrl+href
               
            if site not in  urls: 
                urls.append(site)  
                print(site) 
                # calling it self 
                # scrape(site, baseUrl) 
   
# main function 
if __name__ =="__main__": 
   
    # website to be scrape 
    site="https://www.metatrader4.com/en/trading-platform/help/beginning/autotrading"
    baseUrl = "https://www.metatrader4.com/"
    # calling function 
    scrape(site, baseUrl)

Output

https://www.metatrader4.com/en https://www.metatrader4.com/en/trading-platform https://www.metatrader4.com/en/download https://www.metatrader4.com/en/trading-platform/forex https://www.metatrader4.com/en/trading-platform/orders https://www.metatrader4.com/en/trading-platform/technical-analysis https://www.metatrader4.com/en/trading-platform/alerts-news