accessing html elements BeautifulSoup Python2.7
Question
I'm having trouble getting all of this html's href class attribute value in a list. I'm not sure what I'm doing wrong, I can't even access the reference.
The below is a snippit of what I'm trying to parse:
<!-- <div class="container">
<div class="row">
<div class="col-xs-12 col-md-offset-2 col-md-8 col-md-offset-2">
<div id='location_list'><h2>Browse by location</h2><ol class='suburb_locations'><div class="row"><li class="col-sm-3"><a href="http://www.carnextdoor.com.au/car-rental/nsw/abbotsford-nsw">abbotsford, NSW</a><br><span class="sub_title">0 active owners</span><span class="sub_title">0 active borrowers</span></li><li class="col-sm-3"><a href="http://www.carnextdoor.com.au/car-rental/vic/abbotsford-vic">abbotsford, VIC</a><br><span class="sub_title">0 active owners</span><span class="sub_title">0 active borrowers</span>
I'm trying to get a dict of key=place and the value = number of active borrowers in one list. And a list that contains the href values. My biggest problem is that I'm not able to access any of these siblings. I've tried a lot of things below is a list of a few codes I've been trying:
from bs4 import Beautiful Soup
soup=BeautifulSoup(html,"html5lib")
print soup.find_all('br')
print soup.find_all('div h2 ol li')
print soup.find('li',{'class':"col-sm-3"})
1 answers
solution1
2 ACCPTED 2016-03-08 01:05:44
The problem is the <!-- , if you print soup you will see there is nothing there and when you remove it you get the html.
In [2]: from bs4 import BeautifulSoup
In [3]: soup = BeautifulSoup(html,"lxml")
In [4]: print(soup)
In [5]: soup = BeautifulSoup(html.replace("<!--",""),"lxml")
In [6]: print(soup)
<html><body><div class="container">
<div class="row">
<div class="col-xs-12 col-md-offset-2 col-md-8 col-md-offset-2">
<div id="location_list"><h2>Browse by location</h2><ol class="suburb_locations"><div class="row"><li class="col-sm-3"><a href="http://www.carnextdoor.com.au/car-rental/nsw/abbotsford-nsw">abbotsford, NSW</a><br/><span class="sub_title">0 active owners</span><span class="sub_title">0 active borrowers</span></li><li class="col-sm-3"><a href="http://www.carnextdoor.com.au/car-rental/vic/abbotsford-vic">abbotsford, VIC</a><br/><span class="sub_title">0 active owners</span><span class="sub_title">0 active borrowers</span></li></div></ol></div></div></div></div></body></html>
In [6]: soup.select(".col-sm-3")
Out[6]:
[<li class="col-sm-3"><a href="http://www.carnextdoor.com.au/car-rental/nsw/abbotsford-nsw">abbotsford, NSW</a><br/><span class="sub_title">0 active owners</span><span class="sub_title">0 active borrowers</span></li>,
<li class="col-sm-3"><a href="http://www.carnextdoor.com.au/car-rental/vic/abbotsford-vic">abbotsford, VIC</a><br/><span class="sub_title">0 active owners</span><span class="sub_title">0 active borrowers</span></li>]
In [7]: soup.select(".col-sm-3")[0].text
Out[7]: u'abbotsford, NSW0 active owners0 active borrowers'
I am not sure where you are getting the html from but you need to clean it if you want to parse it.
The conatiner is completely commented outbut you do a replace of just the opening <!-- in the source we can then pull section:
import requests
r = requests.get("http://www.carnextdoor.com.au/find-a-car/")
from bs4 import BeautifulSoup
soup = BeautifulSoup(r.content.replace("<!--",""))
print(soup.select("div #location_list"))
Which gives you:
[<div id="location_list"><h2>Browse by location</h2><ol class="suburb_locations"><div class="row"><li class="col-sm-3"><a href="http://www.carnextdoor.com.au/car-rental/nsw/abbotsford-nsw">abbotsford, NSW</a><br/><span class="sub_title">0 active owners</span><span class="sub_title">0 active borrowers</span></li><li class="col-sm-3"><a href="http://www.carnextdoor.com.au/car-rental/vic/abbotsford-vic">abbotsford, VIC</a><br/><span class="sub_title">0 active owners</span><span class="sub_title">0 active borrowers</span></li><li class="col-sm-3"><a href="http://www.carnextdoor.com.au/car-rental/vic/aberfeldie">aberfeldie, VIC</a><br/><span class="sub_title">0 active owners</span><span class="sub_title">1 active borrower</span></li><li class="col-sm-3"><a href="http://www.carnextdoor.com.au/car-rental/sa/adelaide">adelaide, SA</a><br/><span class="sub_title">0 active owners</span><span class="sub_title">1 active borrower</span></li></div><div class="row"><li class="col-sm-3"><a href="http://www.carnextdoor.com.au/car-rental/act/ainslie">ainslie, ACT</a><br/><span class="sub_title">0 active owners</span><span class="sub_title">1 active borrower</span></li><li class="col-sm-3"><a href="http://www.carnextdoor.com.au/car-rental/vic/aireys-inlet">aireys inlet, VIC</a><br/><span class="sub_title">0 active owners</span><span class="sub_title">1 active borrower</span></li><li class="col-sm-3"><a href="http://www.carnextdoor.com.au/car-rental/vic/airly">airly, VIC</a><br/><span class="sub_title">0 active owners</span><span class="sub_title">1 active borrower</span></li><li class="col-sm-3"><a href="http://www.carnextdoor.com.au/car-rental/vic/airport-west">airport west, VIC</a><br/><span class="sub_title">0 active owners</span><span class="sub_title">1 active borrower</span></li></div><div class="row"><li class="col-sm-3"><a href="http://www.carnextdoor.com.au/car-rental/vic/albert-park">albert park, VIC</a><br/><span class="sub_title">0 active owners</span><span class="sub_title">5 active borrowers</span></li><li class="col-sm-3"><a href="http://www.carnextdoor.com.au/car-rental/sa/aldgate">aldgate, SA</a><br/><span class="sub_title">1 active owner</span><span class="sub_title">2 active borrowers</span></li><li class="col-sm-3"><a href="http://www.carnextdoor.com.au/car-rental/nsw/alexandria">alexandria, NSW</a><br/><span class="sub_title">1 active owner</span><span class="sub_title">53 active borrowers</span></li><li class="col-sm-3"><a href="http://www.carnextdoor.com.au/car-rental/nsw/alexandria-mc">alexandria mc, NSW</a><br/><span class="sub_title">0 active owners</span><span class="sub_title">1 active borrower</span></li></div><div class="row"><li class="col-sm-3"><a href="http://www.carnextdoor.com
And a whole lot more, basically everything of interest in that commented section.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.
Related Question
How to find text with a particular value BeautifulSoup python2.7
accessing a list of urls via Multi-processing in Python2.7
Regex python2.7
CGI with Python2.7
PyMongo with Python2.7
REGEX in python2.7
Multiprocessing in Python2.7
Compute mean of last X elements of a list in Python2.7
dict does not reference elements? Python2.7 changed behavior
How do I pop() elements inside the list in python2.7
Related Tags