使用 Python 逐行写入 CSV

Question

I've written a Python boto3 code to get the average EC2 CPU utilization/day for the last 2 days. Here's the code:

import boto3
import datetime
import csv

accountId = boto3.client('sts').get_caller_identity()['Account']
session = boto3.session.Session()
region = session.region_name
ec2 = session.resource('ec2',region_name=region)
s3 = session.resource('s3')

fields = ['Account' , 'Region' , 'InstanceID' , 'InstanceName']
start = datetime.datetime.today() - datetime.timedelta(days=2)
end = datetime.datetime.today()

instanceId = ''
instanceName = ''
rows = []
filename = 'CPUUtilization.csv'

def get_cpu_utilization(instanceId):
    cw = boto3.client('cloudwatch',region_name=region)
    res = cw.get_metric_statistics(
        Namespace = 'AWS/EC2',
        Period = 86400,
        StartTime = start,
        EndTime = end,
        MetricName = 'CPUUtilization',
        Statistics = ['Average'],
        Unit = 'Percent',
        Dimensions = [
            {
                'Name' : 'InstanceId',
                'Value' : instanceId
            }
        ]
    )
    return res

def lambda_handler(event, context):
    for instance in ec2.instances.all():
        if instance.tags != None:
            for tags in instance.tags:
                if tags['Key'] == 'Name':
                    instanceName = tags['Value']
                    break 
        instanceId = str(instance.id)
        response = get_cpu_utilization(instanceId)
        rows.append([accountId, region, instanceId, instanceName])

        for r in response['Datapoints']:
            day = r['Timestamp'].date()
            week = day.strftime('%a')
            avg = r['Average']
            day_uti = ' '.join([str(day),week])
            fields.append(day_uti)
            rows.append([avg])

    with open("/tmp/"+filename, 'w+') as csvfile:
        csvwriter = csv.writer(csvfile)
        csvwriter.writerow(fields)
        csvwriter.writerows(rows)
    csvfile.close()

    s3.Bucket('instances-cmdb').upload_file('/tmp/CPUUtilization.csv', 'CPUUtilization.csv')

The output written to the CSV file is like this:

The average CPU utilization value is printed in the A3 cell, but this has to be printed/written to E2 cell under the date. And all the subsequent days to be written to 1st row and the corresponding values should go to 2nd row, cell by cell, under their respective dates.

How can I achieve this?

I have a couple of other questions related to AWS CloudWatch metrics.

1. This particular instance was in stopped state the whole day (1st April 2022). Still this Lambda function is giving some CPU utilization value on that day. When I checked for the same from the console, I don't see any data. How is this possible? Am I making any mistake?

2. When I ran this function multiple times, I got different CPU utilization values. The above attached image was from 1st execution (Avg CPU utilization=0.110935...). Below is the result from 2nd execution

Here the avg CPU utilization for the same instance on the same day is different(0.53698..) from previous result. Is this mistake from my side or what?

Please help.

NOTE: There is only one instance in my account and it was in stopped state the whole day (1st April 2022) and started only on 2nd April 2022 at around 8:00PM IST.

Answer 1

You need to rethink your logic for adding columns for each datapoint returned.

The row list contains one entry per row. It starts with this:

rows.append([accountId, region, instanceId, instanceName])

That creates one entry in the list that is a list with four values.

Later, the code attempts to add another column with:

rows.append([avg])

This results in rows having the value of [[accountId, region, instanceId, instanceName], [avg]] .

This is adding another row , which is why it is appearing in the CSV file as a separate line. Rather than adding another row, the code needs to add another entry in the existing row .

The easiest way to do this would be to save the row in a list and only add the 'row' once you have all the information for the row .

So, you could replace this line:

rows.append([accountId, region, instanceId, instanceName])

with:

current_row = [accountId, region, instanceId, instanceName]

And you could later add to it with:

current_row.append(avg)

Then, after the for loop has completed adding all the columns, it can be stored with:

rows.append(current_row)

Also, be careful with this line:

fields.append(day_uti)

It is adding the date to the fields list, but if there is more than one instance, each instance will add an entry. I presume you want them to be the same date, so it won't work out like you expect.