How to use NextToken in Boto3

Question

The below-mentioned code is created for exporting all the findings from the security hub to an S3 bucket using lambda functions. The filters are set for exporting only CIS-AWS foundations benchmarks. There are more than 20 accounts added as the members in security hub. The issue that I'm facing here is even though I'm using the NextToken configuration. The output doesn't have information about all the accounts. Instead, it just displays any one of the account's data randomly.

Can somebody look into the code and let me know what could be the issue, please?

import boto3
import json
from botocore.exceptions import ClientError
import time
import glob
 
client = boto3.client('securityhub')
s3 = boto3.resource('s3')
 
storedata = {}
_filter = Filters={
'GeneratorId': [
{
'Value': 'arn:aws:securityhub:::ruleset/cis-aws-foundations-benchmark',
'Comparison': 'PREFIX'
}
],
}
 
def lambda_handler(event, context):
    response = client.get_findings(
    Filters={
        'GeneratorId': [
            {
                'Value': 'arn:aws:securityhub:::ruleset/cis-aws-foundations-benchmark',
                'Comparison': 'PREFIX'
            },
        ],
    },
    )
    results = response["Findings"]
    while "NextToken" in response:
        response = client.get_findings(Filters=_filter,NextToken=response["NextToken"])
        results.extend(response["Findings"])
        storedata = json.dumps(response)
    print(storedata)
 
    save_file = open("/tmp/SecurityHub-Findings.json", "w")
    save_file.write(storedata)
    save_file.close()
 
    for name in glob.glob("/tmp/*"):
      s3.meta.client.upload_file(name, "xxxxx-security-hubfindings", name)

TooManyRequestsException error is also getting now.

Answer 1

The problem is in this code that paginates the security findings results:

while "NextToken" in response:
    response = client.get_findings(Filters=_filter,NextToken=response["NextToken"])
    results.extend(response["Findings"])
    storedata = json.dumps(response)

print(storedata)

The value of storedata after the while loop has completed is the last page of security findings, rather than the aggregate of the security findings.

However, you're already aggregating the security findings in results , so you can use that:

save_file = open("/tmp/SecurityHub-Findings.json", "w")
save_file.write(json.dumps(results))
save_file.close()