从python中的一个html表爬取数据

Question

I am a beginner in web crawling and I need help in getting the values from the table. I have got all the required fields (LOCATION,DATE,SUMMARY,DEADLINE). What I want is the Summary is having a url to another page. I want that url to get appended along with the other fields like (LOCATION,DATE,SUMMARY,DEADLINE, URL )

This is the website

This is my code so far. But its not working

import requests as rq
from bs4 import BeautifulSoup
import pandas as pd

url = 'https://www.tendersinfo.com/global-information-technology-tenders-{}.php'

amount_of_pages = 2 #5194 
rows = []

for i in range(1,amount_of_pages):
    response = rq.get(url.format(i))


    if response.status_code == 200:
        soup = BeautifulSoup(response.text,'html.parser')
        table = soup.find('table',{'id':'datatable'})

        headers = []

        for th in table.find("tr").find_all("th"):
           headers.append(th.text.strip())

        for tr in table.find_all("tr")[1:]:
            cells = []
            tds = tr.find_all("td")

            if len(tds) == 0:
                ths = tr.find_all("th")

                for th in ths:
                    cells.append(th.text.strip())
            else:
                for td in tds:
                    cells.append(td.text.strip())
                    cells.append('https://www.tendersinfo.com/' + td.find('a')['href'])

            rows.append(cells)   

Answer 1

Here you go, I just re-coded the majority of it.

import requests as rq
from bs4 import BeautifulSoup
import pandas as pd

location = []
posted_date = []
summary = []
deadline = []

url = 'https://www.tendersinfo.com/global-information-technology-tenders-{}.php'

amount_of_pages = 10 # Max is 5194 currently
rows = []

for i in range(1,amount_of_pages):
    response = rq.get(url.format(i))
    if response.status_code == 200:
        soup = BeautifulSoup(response.text,'html.parser')
        table = soup.find('table',{'id':'datatable'})
        headers = []
        for th in table.find("tr").find_all("th"):
           headers.append(th.text.strip())
        for tr in table.find_all("tr")[1:]:
            cells = []
            tds = tr.find_all("td")
            if len(tds) == 0:
                ths = tr.find_all("th")
                for th in ths:
                    cells.append(th.text.strip())
            else:
                for td in tds:
                    cells.append(td.text.strip())
            rows.append(cells)

pd.DataFrame(rows, columns=headers).to_csv(r"C:\Users\HP\Desktop\Web Scraping (RFP's)\RFP_SCRAPED_DATA.csv", index=False)

Answer 2

Since your using pandas, why not use read_html which return's extracted tables as a list of DataFrames.

>>> tables = pd.read_html("https://www.tendersinfo.com/global-information-technology-tenders.php")

>>> tables[1]

  LOCATION         DATE                                        SUMMARY     DEADLINE
0    India  21-May-2020  Liquid Crystal Display Lcd Panel Or Monitors.  01-Jun-2020
1    India  21-May-2020                          Random Access Memory.  01-Jun-2020
2    India  21-May-2020         Supply Of Analog Transceiver-handheld.  01-Jun-2020
3    India  21-May-2020                   Supply Of Computer Printers.  01-Jun-2020
4    India  21-May-2020                                 All In One Pc.  01-Jun-2020

Answer 3

You get the table easily using pd.read_html and save this data into csv file using df.to_csv() .

import pandas as pd

url = "https://www.tendersinfo.com/ajax_all_new_search.php?country=information-technology&increment=1&%20select=500&%20total=259655&%20search_id=19906&%20order=id&%20imagevalue=1"

df = pd.read_html(url)[0]

df.to_csv("RFP_SCRAPED_DATA.csv", index=False)