why do I keep getting 'NoneType' object has no attribute 'a' in django app?

Question

I'm confused as to why I'm getting

'NoneType' object has no attribute 'a' 

This is the html structure I am scraping

<section class ="videos"
<section class="box">
<a href="/videos/video.php?v=wshhH0xVL2LP4hFb0liu" class="video-box">
    <img src="http://hw-static.exampl.net/.jpg" width="222" height="125" alt="">
</a>
<strong class="title"><a href="/videos/video.php?v=wshhH0xVL2LP4hFb0liu">Teen "Allegedly" </a></strong>
<div>
    <span class="views">11,323</span> 
    <span class="comments"><a href="http://www.example.net/v" data-disqus-identifier="94137">44</a></span>
</div>

in my Django app. If I do this

 html = requests.get(vlad_url)
        soup = BeautifulSoup(html.text, 'html.parser')
        divs = soup.find('section', 'videos')

        img = divs.find('img').get('src')
        text = divs.strong.a.text
        link = divs.a.get('href')

context = {
    "ref": link,
    "src": img,
    "txt": text,
}

in my views. And This in my template

{{ref}}
{{src}}
{{txt}}

I will get a single result for each. But When I try to loop through them like this

def get_vlad(url):
        html = requests.get(url, headers=headers)
        soup = BeautifulSoup(html.text, 'html.parser')
        divs = soup.findAll('section', 'box')

        entries = [{'text': div.strong.a.text,
                    'link': div.a.get('href'),
                    'img': div.find('img').get('src')
                       } for div in divs]
        return entries

I get that Nonetype error which is strange because it does exist. It's also strange for the reason I have another loop similar to this one that works

    def get_data(uri):

        html = requests.get(uri, headers=headers)
        soup = BeautifulSoup(html.text, "html.parser")
        divs = soup.findAll('div', 'thumbnail')
        entries = [{'text': div.text,
                    'href': div.find('a').get('href'),
                    'src': div.find('img').get('src')
                    } for div in divs][:6]
        return entries

and this is the html structure it works on

 <div class="col-xs-12 col-md-4" id="split">
      <div class="thumbnail thumb">

             <h6 id="date">May 6, 2016</h6>

            <img src="http://www.paraguayhits.com/wp-content/uploads/2015/11/Almighty-Ft.-N%CC%83engo-Flow-Por-Si-Roncan-660x330.jpg" class="img-responsive post">


        <div style="border-bottom: thin solid lightslategray; padding-bottom: 15px;"></div>

        <div class="caption" id="cap">
            <a href="/blog/almighty-por-si-roncan-ft-nengo-flow-official-video/">
                <h5 class="post-title" id="title">Almighty - Por Si Roncan (ft. Ñengo Flow) [Official Video]</h5>
            </a>





            <p>
                <a href="/blog/76/delete/" class="btn" role="button">delete</a>
                <a href="/blog/almighty-por-si-roncan-ft-nengo-flow-official-video/edit/" class="btn" role="button">edit</a>
            </p>

        </div>
    </div>

what's the difference between the two? how can i loop through my results

Answer 1

The html is broken, the section tags are a mess, I have had success using html5lib for parsing badly broken html with bs4:

In [21]: h = """<section class="videos"
   ....: <section class="box">
   ....: <a href="/videos/video.php?v=wshhH0xVL2LP4hFb0liu" class="video-box">
   ....:     <img src="http://hw-static.exampl.net/.jpg" width="222" height="125" alt="">
   ....: </a>
   ....: <strong class="title"><a href="/videos/video.php?v=wshhH0xVL2LP4hFb0liu">Teen "Allegedly" </a></strong>
   ....: <div>
   ....:     <span class="views">11,323</span>
   ....:     <span class="comments"><a href="http://www.example.net/v" data-disqus-identifier="94137">44</a></span>
   ....: </div>"""

In [22]: from bs4 import BeautifulSoup

In [23]: soup = BeautifulSoup(h, 'html5lib')

In [24]: divs = soup.select_one('section.videos')

In [25]: img = divs.find('img').get('src')

In [26]: text = divs.strong.a.text

In [27]: link = divs.a.get('href')

In [28]: img
Out[28]: u'http://hw-static.exampl.net/.jpg'

In [29]: text
Out[29]: u'Teen "Allegedly" '

In [30]: link
Out[30]: u'/videos/video.php?v=wshhH0xVL2LP4hFb0liu'

The correct html should look something like:

<section class ="videos">
        <section class="box">
            <a href="/videos/video.php?v=wshhH0xVL2LP4hFb0liu" class="video-box">
                <img src="http://hw-static.exampl.net/.jpg" width="222" height="125" alt="">
            </a>
            <strong class="title"><a href="/videos/video.php?v=wshhH0xVL2LP4hFb0liu">Teen "Allegedly" </a></strong>
        </section>
        <div>
            <span class="views">11,323</span>
            <span class="comments"><a href="http://www.example.net/v" data-disqus-identifier="94137">44</a></span>
        </div>
</section