Appending API results in the existing CSV file in a right format (New header + API results)

Question

Currently I'm dealing with 50,000 rows of CSV table. Here is the sample CSV table I'm inputting in an API:

My input

API then processes address in each row and gives me a corresponding coordinates (latitude & longitude)

My problem is that I need to append these new headers and values in the existing table. (Say at row[22])

Here is my Python query below:

Python query

import requests
import json
import pandas as pd
import numpy as np
import csv
import sys
from geocodio import GeocodioClient
import re

client = GeocodioClient('506be11563600404eb83151e40bb0f11ef06f3b')


# Input - CSV


df=pd.read_csv(r"C:\users\testu\documents\travis_50000_melissa_joined_dropna - Copy2.csv",delimiter=',', na_values="nan")


with open(r"C:\users\testu\documents\travis_50000_melissa_joined_dropna - Copy2.csv", 'a', newline='') as fp:
   fieldnames = ["latitude","longitude","coordinates"]
   writer = csv.DictWriter(fp, fieldnames=fieldnames)
   writer.writeheader()

   # Iterating requests for each row
   for row in df.itertuples():
        output = client.geocode(str(row.addressline1) + ', ' + str(row.city) + ', ' + str(row.state) + ', ' + str(row.postalcode)).coords
        cord = '(' + str(output[0]) + ', '+ str(output[1]) + ')'

        writer.writerow({'latitude': output[0], 'longitude': output[1], 'coordinates': cord})
        print(output)

As you can see in the image below, I'm getting API results in a new row.

My current output

How can I get the desired CSV output as shown in the image below?

Note: No Pandas please! I need to use CSVwriter to solve this issue.

Desired output

Answer 1

You need to append the three new fields to each row , so you need to merge row with the three new fields. That's gonna be a bit messy in Pandas, so allow me to rephrase pd.read_csv() using the csv module:

import csv

with open("test.csv") as in_file, open("test_out.csv", "w") as out_file:
    csv_in = csv.DictReader(in_file, delimiter=",")
    headers = csv_in.fieldnames + ["lat", "lon", "coord"]
    csv_out = csv.DictWriter(out_file, fieldnames=headers)
    csv_out.writeheader()
    for row in csv_in:
        output = client.geocode("{}, {}, {}, {}".format(
            row["addressline1"],
            row["city"],
            row["state"],
            row["postalcode"]
        ))
        row["lat"] = output[0]
        row["lon"] = output[1]
        row["coord"] = "({} {})".format(output[0], output[1])
        csv_out.writerow(row)

I would strongly recommend you create a new file and don't just overwrite the old file.