beautifulsoup - 无法从列表中获取信息

Question

Total python amateur here. I've taken a few classes for some surface-level stuff but haven't come across solutions to this problem for a new website I'm trying to scrape (kijiji.ca for anyone wondering). I'm trying to pull down info on rental housing for some PhD dissertation-related work. Inspecting a sample page, I'm finding that some key info I need all has the same class. For example:

 <div class="titleAttributes-2381855425"> <li class="noLabelAttribute-1492730675"><svg class="icon-459822882 attributeIcon-1499443538 attributeIcon__condensed-4247835132" focusable="false" height="100%" role="img" width="100%"><use xlink:href="#icon-attributes-unittype"></use></svg><span class="noLabelValue-3861810455">Condo</span></li> <li class="noLabelAttribute-1492730675"><svg class="icon-459822882 attributeIcon-1499443538 attributeIcon__condensed-4247835132" focusable="false" height="100%" role="img" width="100%"><use xlink:href="#icon-attributes-numberbedrooms"></use></svg><span class="noLabelValue-3861810455">Bedrooms: 2</span></li> <li class="noLabelAttribute-1492730675"><svg class="icon-459822882 attributeIcon-1499443538 attributeIcon__condensed-4247835132" focusable="false" height="100%" role="img" width="100%"><use xlink:href="#icon-attributes-numberbathrooms"></use></svg><span class="noLabelValue-3861810455">Bathrooms: 1</span></li> </div>

I'm trying to get each piece of info, but when I run my code, it just shows up with nothing.

 def getDetails(urls): urls = urls[10:] print(len(urls)) i =0; try: for url in urls: print(url) listDetails = "" listDetailsTwo = [] url = url.rstrip('\\n') response = requests.get(url) soup = BeautifulSoup(response.text, "html.parser") try: infobar = soup.select_one("span[class*=noLabelValue-3861810455]").text infobar.append(infobar) print("Scraping listing : ",str(i))

(I know, my code must look like an absolute mess, but again, I'm a total amateur.) I know I must need to use something other than soup.select_one, but after a few days of trying, I'm really getting nowhere. Any help would be much appreciated!

Thanks!

Answer 1

You probably wantsoup.find_all() :

L = soup.find_all('span', {'class':'noLabelValue-3861810455'})
for item in L:
    print(item.text)

Prints:

Condo
Bedrooms: 2
Bathrooms: 1

Answer 2

Try this.

from simplified_scrapy import SimplifiedDoc,req,utils
html='''
<div class="titleAttributes-2381855425">
  <li class="noLabelAttribute-1492730675"><svg class="icon-459822882 attributeIcon-1499443538 attributeIcon__condensed-4247835132" focusable="false" height="100%" role="img" width="100%"><use xlink:href="#icon-attributes-unittype"></use></svg><span class="noLabelValue-3861810455">Condo</span></li>
  <li
    class="noLabelAttribute-1492730675"><svg class="icon-459822882 attributeIcon-1499443538 attributeIcon__condensed-4247835132" focusable="false" height="100%" role="img" width="100%"><use xlink:href="#icon-attributes-numberbedrooms"></use></svg><span class="noLabelValue-3861810455">Bedrooms: 2</span></li>
    <li
      class="noLabelAttribute-1492730675"><svg class="icon-459822882 attributeIcon-1499443538 attributeIcon__condensed-4247835132" focusable="false" height="100%" role="img" width="100%"><use xlink:href="#icon-attributes-numberbathrooms"></use></svg><span class="noLabelValue-3861810455">Bathrooms: 1</span></li>
</div>'''
doc = SimplifiedDoc(html)
spans = doc.selects('span.noLabelValue-3861810455>text()')
print (spans)
spans = doc.selects('div.titleAttributes-2381855425>li.noLabelAttribute-1492730675').select('span.noLabelValue-3861810455>text()')
print (spans)
utils.save2csv('test.csv',[spans])

Result:

['Condo', 'Bedrooms: 2', 'Bathrooms: 1']
['Condo', 'Bedrooms: 2', 'Bathrooms: 1']