无法通过BeautifulSoup / LXML解析HTML

Question

I have an HTML page and I want to find some items of it. I am finding it hard to apply beautifulsoup or lxml

HTML page:

<li class="context-card">
    <div class="episode" data-id="t1">
        <span class="av-play">Title to scrape</span>
    </div>
</li>
<li class="context-card">
    <div class="episode" data-id="t2">
        <span class="av-play">Title2 to scrape</span>
    </div>
</li>
<li class="context-card">
    <div class="episode" data-id="t3">
        <span class="av-play">Title3 to scrape</span>
    </div>
</li>

How to get all these 3 Ids and titles in a different dictionary within a list

[{'id':'t1', 'title': 'Title to scrape'}, {'id':'t2', 'title': 'Title2 to scrape'}, {'id':'t3', 'title': 'Title3 to scrape'}]

Answer 1

All the titles and IDs you need are located inside the <span> tag with the class="episode" attribute. So, your job is to iterate over all of those tags and get the 'data-id' of the div tag and the text of its inner span tag.

Code:

html = '''
<li class="context-card">
    <div class="episode" data-id="t1">
        <span class="av-play">Title to scrape</span>
    </div>
</li>
<li class="context-card">
    <div class="episode" data-id="t2">
        <span class="av-play">Title2 to scrape</span>
    </div>
</li>
<li class="context-card">
    <div class="episode" data-id="t3">
        <span class="av-play">Title3 to scrape</span>
    </div>
</li>
'''
soup = BeautifulSoup(html, 'lxml')

title_list = []
for ep in soup.find_all('div', class_='episode'):
    curr_dict = {'id': ep['data-id'], 'title': ep.span.text}
    title_list.append(curr_dict)

print(title_list)

Output:

[{'id': 't1', 'title': 'Title to scrape'},
 {'id': 't2', 'title': 'Title2 to scrape'},
 {'id': 't3', 'title': 'Title3 to scrape'}]

Or, the same can be done using a list comprehension:

title_list = [{'id': ep['data-id'], 'title': ep.span.text} for ep in soup.find_all('div', class_='episode')]