Save Scraped Data as CSV file?

Question

I am trying to scrape data from a link that contains JSON data and this is the code:

    import requests
    import json

    parameters = ['a:1','a:2','a:3','a:4','a:3','a:4','a:5','a:6','a:7','a:8','a:9','a:10',]

    for item in parameters:
        key, value = item.split(':')[0], item.split(':')[1]
        url = "https://xxxx.000webhostapp.com/getNamesEnc02Motasel2.php?keyword=%s&type=2&limit=%s" %(key, value)
        r = requests.get(url)
        cont = json.loads(r.content)
        print(cont)

And the output be like

[{'name': 'Absz', 'phone': '66343212'}, {'name': 'ddd ', 'phone': '545432211'}, {'name': 'ezd' 'phone':'54856886'}]

I want to store all the data in a CSV file.

How can I do this?

Also, As you can see I am using parameters list to do multi requests but I think there is a way that I can loop the limit parameter from 1 to 200 without typing every single keyword and number in parameters.

Thanks in advance.

Answer 1

Try the below code it will create csv row wise:

import csv
import json

header = ["name","phone"]

for item in range(1,200): 
    key, value = 'a', item # Generating key and value from range 1 --> 200
    url = "https://xxxx.000webhostapp.com/getNamesEnc02Motasel2.php?keyword=%s&type=2&limit=%s" %(key, value)
    r = requests.get(url)
    cont = json.loads(r.content)
    print(cont)



with open('people.csv', 'a') as writeFile:
    writer = csv.writer(writeFile)
    writer.writerow(header)
    for a_row in cont:
        writer.writerow([a_row["name"],a_row["phone"]]) # To write name and phone

Hope this answers your question!!

Answer 2

import requests
import json
import pandas as pd

parameters = ['a:1','a:2','a:3','a:4','a:3','a:4','a:5','a:6','a:7','a:8','a:9','a:10']

results = pd.DataFrame()
for item in parameters:
    key, value = item.split(':')
    url = "https://xxxx.000webhostapp.com/getNamesEnc02Motasel2.php?keyword=%s&type=2&limit=%s" %(key, value)
    r = requests.get(url)
    cont = json.loads(r.content)
    temp_df = pd.DataFrame(cont)
    results = results.append(temp_df)

results.to_csv('path/to/filename.csv', index=False)