简体   繁体   English

Python Web Scraping-将数据保存为CSV

[英]Python Web Scraping - Save data in CSV

I am trying to save the data scraped from URLs such as " https://www.holidify.com/places/shimla/mall-road-shimla-sightseeing-3502.html ". 我正在尝试保存从诸如“ https://www.holidify.com/places/shimla/mall-road-shimla-sightseeing-3502.html ”之类的URL抓取的数据。 When saving the data in a csv file only the data from last url from the range gets saved in the csv file. 将数据保存在csv文件中时,只有来自范围中最后一个URL的数据才保存在csv文件中。 I need that data from all of the URLs gets saved in the csv file. 我需要将所有URL中的数据保存在csv文件中。

pages = []
for i in range(1, 10, 1):
    url = "https://www.holidify.com/places/shimla/mall-road-shimla-sightseeing-350" + str(i) + '.html'
    pages.append(url)
    for item in pages:
        page = requests.get(item)
        soup = BeautifulSoup(page.text, 'html.parser')
        Place = list(soup.find(class_="col-md-10 col-xs-10 nopadding"))[1].get_text()
        City = list(soup.find_all(class_="smallerText"))[1].get_text()
        State = list(soup.find_all(class_="smallerText"))[2].get_text()
        Country = list(soup.find_all(class_="smallerText"))[3].get_text()
        About = list(soup.find_all(class_="biggerTextOverview"))[0].get_text()
        more_About = list(soup.find_all(class_="objHeading smallerText"))[0].get_text()
        Weather = soup.find(class_="currentWeather").get_text()
        demo = pd.DataFrame({ "Place": Place, "City": City, "State": State, "Country": Country, "About": About,"More About Places": more_About}, index=[0])
        demo.to_csv('demo.csv', index=False, encoding='utf-8')

您需要将数据追加到该文件中

demo.to_csv('demo.csv', index=False, encoding='utf-8', mode = 'a')

如@Umair所建议,将数据追加到数据帧中,并将命令demo.to_csv('demo.csv',index = False,encoding ='utf-8')置于循环外部。

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM