使用Python仅收集网页中的href的第一级

Question

I need to retrieve only the 1st level in a href of a website. For example: http://www.example.com/ is the website that I need to open and read.I opened the page and collected the href's and I obtained all the links like /company/organization, /company/globallocations, /company/newsroom, /contact, /sitemap and so on.

Below is the python code.

req = urllib2.Request(domain)
response = urllib2.urlopen(req)
soup1 = BeautifulSoup(response,'lxml')
for link in soup1.find_all('a',href = True):
     print link['href']

My desired output is,

/company, /contact, /sitemap for the website www.example.com

Kindly help and suggest me a solution.

Answer 1

The first level concept is not clear, if you believe href links with one / is a first level, just simply count how many / in the href text, and decide keep it or drop it.

If we consider the web page point of view, all links in the home page, should be considered as first level . In this case, you may need to create a level counter to count how many levels / how deep your crawler goes into, and stop at certain level.

Hope that helps.