How to export all the Details in the Div using Beautiful soup python to excel/csv?

Question

i am Newbie to Soup/python and i am trying to find the data. my website Structure look like this.

and if i open the Divclass border class it look like this.(image below)

I have done something like this:

for  P in soup.find_all('p', attrs={'class': 'bid_no pull-left'}) :
  print(P.find('a').contents[0])

a Div structure look like
there are around 10 div in each page
this in which i want to extract the Items,Quantity Require,Bid number,End date . Please help me

<div class="border block " style="display: block;">
    <div class="block_header">
        <p class="bid_no pull-left"> BID NO: <a style="color:#fff !important" href="/showbidDocument/1844736">GEM/2020/B/763154</a></p> 
        <p class="pull-right view_corrigendum" data-bid="1844736" style="display:none; margin-left: 10px;"><a href="#">View Corrigendum</a></p>

         <div class="clearfix"></div>
    </div>

    <div class="col-block">
        <p><strong style="text-transform: none !important;">Item(s): </strong><span>Compatible Cartridge</span></p>
        <p><strong>Quantity Required: </strong><span>8</span></p>

        <div class="clearfix"></div>
    </div>
    <div class="col-block">
        <p><strong>Department Name And Address:</strong></p>
        <p class="add-height">
            Ministry Of Railways<br> Na<br> South Central Railway N/a
        </p>
        <div class="clearfix"></div>
    </div>
    <div class="col-block">
        <p><strong>Start Date: </strong><span>25-08-2020 02:54 PM</span></p>
        <p><strong>End Date: </strong><span>04-09-2020 03:00 PM</span></p>
        <div class="clearfix"></div>

    </div>


    <div class="clearfix"></div>
</div>

Error image

Answer 1

Try the below approach using requests and beautiful soup . I have created the script with the URL which is fetched from website and then creating a dynamic URL to traverse each and every page to get the data.

What exactly script is doing:

1. First script will create a URL where page_no query string parameter will increment by 1 upon completion of each traversal.

2. Requests will get the data from the created URL using get method which will then pass to beautiful soup to parse HTML structure using lxml .

3. Then from the parsed data script will search for the div where data is actually present.

4. Finally looping on all the div text data one by one for each page.

    ```python import requests from urllib3.exceptions import InsecureRequestWarning requests.packages.urllib3.disable_warnings(InsecureRequestWarning) from bs4 import BeautifulSoup as bs def scrap_bid_data(): page_no = 1 #initial page number while True: print('Hold on creating URL to fetch data...') URL = 'https://bidplus.gem.gov.in/bidlists?bidlists&page_no=' + str(page_no) #create dynamic URL print('URL cerated: ' + URL) scraped_data = requests.get(URL,verify=False) # request to get the data soup_data = bs(scraped_data.text, 'lxml') #parse the scraped data using lxml extracted_data = soup_data.find('div',{'id':'pagi_content'}) #find divs which contains required data if len(extracted_data) == 0: # **if block** which will check the length of extracted_data if it is 0 then quit and stop the further execution of script. break else: for idx in range(len(extracted_data)): # loops through all the divs and extract and print data if(idx % 2 == 1): #get data from odd indexes only because we have required data on odd indexes bid_data = extracted_data.contents[idx].text.strip().split('\\n') print('-' * 100) print(bid_data[0]) #BID number print(bid_data[5]) #Items print(bid_data[6]) #Quantitiy Required print(bid_data[10] + bid_data[12].strip()) #Department name and address print(bid_data[16]) #Start date print(bid_data[17]) #End date print('-' * 100) page_no +=1 #increments the page number by 1 scrap_bid_data() ```

Actual Code

Output image