Python: extract certain class separately by using bs4

Question

<div class="michelinKeyBenefitsComp">
    <section id="benefit-one-content">
        <div class="inner">
            <div class="col">
                <h4 class="h-keybenefits">Banana is yellow.</h4>
                <div class="content">
                    <p>Yellow is my favorite color.</p>
                    <p>&nbsp;</p>
                    <p>I love Banana.</p>
                </div>
            </div>
        </div>
    </section>      
    <section id="benefit-two-content">
        <div class="inner">
            <div class="col">
                <h4 class="h-keybenefits">Apple is red.</h4>
                <div class="content"><p>Red is not my favorite color.</p>
                    <p>&nbsp;</p>
                    <p>I don't like apple.</p>
                </div>
            </div>
        </div>
    </section>
</div>

I know how to extract all the text I want from this HTML. Here is my code:

for item in soup.find('div', {'class' : 'michelinKeyBenefitsComp'}):
    try:
        for tex in item.find_all('div', {'class' : 'col'}):
            print(tex.text)
    except: 
        pass

But what i would like to do is extract the content separately, so I can save them separately. The result is expected like this:

Banana is yellow.
Yellow is my favorite color.
I love Banana.
#save first

Apple is red.
Red is not my favorite color.
I don't like apple.
#save next 

By the way, in this case, there are only 2 paragraph, but in other cases, there are probably three or more paragraphs. How can I extract them without knowing how many paragraphs they have? TIA

Answer 1

May be you should try this way for extracting text, you have div with unique_id, but for selecting section text inside it you can use classes for properly select text from particular div,

from bs4 import BeautifulSoup
text = """
    <div class="michelinKeyBenefitsComp">
    <section id="benefit-one-content">
        <div class="inner">
            <div class="col">
                <h4 class="h-keybenefits">Banana is yellow.</h4>
                <div class="content">
                    <p>Yellow is my favorite color.</p>
                    <p>&nbsp;</p>
                    <p>I love Banana.</p>
                </div>
            </div>
        </div>
    </section>      
    <section id="benefit-two-content">
        <div class="inner">
            <div class="col">
                <h4 class="h-keybenefits">Apple is red.</h4>
                <div class="content"><p>Red is not my favorite color.</p>
                    <p>&nbsp;</p>
                    <p>I don't like apple.</p>
                </div>
            </div>
        </div>
    </section>
</div>
"""

soup = BeautifulSoup(text, 'html.parser')
main_div = soup.find('div', class_='michelinKeyBenefitsComp')

for idx, div in enumerate(main_div.select('section > div.inner > div.col')):
    with open('file_'+str(idx)+'.txt', 'w', encoding='utf-8') as f:
        f.write(div.get_text())

#Output in separate file: file_1.txt> Banana is yellow.
                                    # Yellow is my favorite color.
                                    # I love Banana.

Answer 2

This should help.

from bs4 import BeautifulSoup
import re

soup = BeautifulSoup(html, "html.parser")
for i in soup.find_all("section", {"id": re.compile("benefit-[a-z]+-content")}): 
    with open(i["id"]+".txt", "a") as outfile:    #Create filename based on section ID and write.
        outfile.write("\n".join([i for i in i.text.strip().split("\n") if i.strip()]) + "\n\n")