how to write the scraped data in the csv fromat?

Question

Hello i'm new to python and I am don't figure out how to convert scraped data into csv format. here's my program

import requests
import urllib.request

from bs4 import BeautifulSoup
import pandas



url = 'https://menupages.com/restaurants/ny-new-york/2'
response = requests.get(url)
soup = BeautifulSoup(response.text, "html.parser")


all_links = soup.find_all("a")
for link in all_links:
    print(link.get("href"))
    rows = soup.find_all('tr')
    print(rows[:10])

it scraped me my desired output and i want to save my output in the csv file.Anyone please help

Answer 1

You can find the following example in the python csv documentation.

import csv
with open('eggs.csv', 'w', newline='') as csvfile:
    spamwriter = csv.writer(csvfile, delimiter=' ',
                            quotechar='|', quoting=csv.QUOTE_MINIMAL)
    spamwriter.writerow(['Spam'] * 5 + ['Baked Beans'])
    spamwriter.writerow(['Spam', 'Lovely Spam', 'Wonderful Spam'])

As you can see all you need to do is to convert a row to a list and then pass it to the writerow method.

Answer 2

You can store list of your scraped links in python list and then by creating pandas DataFrame create a csv file.

import requests
import urllib.request

from bs4 import BeautifulSoup
import pandas


url = 'https://menupages.com/restaurants/ny-new-york/2'
response = requests.get(url)
soup = BeautifulSoup(response.text, "html.parser")


all_links = soup.find_all("a")
list_links = []
for link in all_links:
    list_links.append(link.get("href"))
    rows = soup.find_all('tr')

df = pandas.DataFrame({'WebLinks':list_links})
df.to_csv('/home/stackoverflow/links.csv', index=0)

Output of file

WebLinks
https://menupages.com/
https://menupages.com/
https://menupages.com/restaurants/cities
https://menupages.com/info/about-us
https://menupages.com/info/contact-us