using pandas to_csv to write the result to a csv

Question

Below is my code which gets the instance name from excel sheet and searches for the instance name in aws account and gets the instance id. So that process is working fine. now i wanted to write the data to csv say like in a csv there has to be two columns with names like instancename and instanceId and the data should be printed in respective columns. please help me in getting the desired output. the csv is overlapped with the data and once opening the csv i can just see the last values in it because my script overrides the previous results.

import pandas as pd
from pandas import ExcelWriter
from pandas import ExcelFile
import numpy as np
import os
import boto3
client = boto3.client('ec2')
THIS_FOLDER = os.path.dirname(os.path.abspath(__file__))
my_file = os.path.join(THIS_FOLDER, 'example.xlsx')

df = pd.read_excel(my_file, sheet_name='Sheet2')

list1 = df['EC2NAMES']
print(list1)
client = boto3.client('ec2')
for names in list1:
    custom_filter = [{
        'Name':'tag:Name', 
        'Values': [names]}]
    print(names)
    instances = client.describe_instances(Filters=custom_filter)
    for instance in instances['Reservations']:
        for key in instance["Instances"]:
            x = key['InstanceId']
            print(x)
            data = pd.DataFrame({'A' : [names],'B' : [x]})
            data.to_csv('df111111.csv')

Expected output:

Instancename  InstanceID
testinstance  123456
testinstance1 12345656312
testinstance2 12345657237

Actual output:

Instancename  InstanceID
testinstance2 12345657237

Answer 1

You can use pandas concat or append but best way is to store the data into list and in the end make a dataframe of that and save it as csv.

import pandas as pd
from pandas import ExcelWriter
from pandas import ExcelFile
import numpy as np
import os
import boto3
client = boto3.client('ec2')
THIS_FOLDER = os.path.dirname(os.path.abspath(__file__))
my_file = os.path.join(THIS_FOLDER, 'example.xlsx')

df = pd.read_excel(my_file, sheet_name='Sheet2')

list1 = df['EC2NAMES']
print(list1)
client = boto3.client('ec2')
data = []
for names in list1:
    custom_filter = [{
        'Name':'tag:Name', 
        'Values': [names]}]
    print(names)
    instances = client.describe_instances(Filters=custom_filter)
    for instance in instances['Reservations']:
        for key in instance["Instances"]:
            x = key['InstanceId']
            print(x)
            data.append([names, x])
pd.DataFrame(data, colums=['A','B']).to_csv('df111111.csv')

Answer 2

Your code makes a new 'data' variable for every iteration of the for loop. My attempt is making a blank data variable before starting the loop. Add a new piece to the dataframe each loop, and once you're out of the loop you save it to csv

data = pd.DataFrame()
list1 = df['EC2NAMES']
print(list1)
client = boto3.client('ec2')
for names in list1:
    custom_filter = [{
        'Name':'tag:Name', 
        'Values': [names]}]
    print(names)
    instances = client.describe_instances(Filters=custom_filter)
    for instance in instances['Reservations']:
        for key in instance["Instances"]:
            x = key['InstanceId']
            print(x)
            data = data.append(pd.DataFrame({'A' : [names],'B' : [x]}))
data.to_csv('df111111.csv')