How can I go through and parse ALL LOGS on my AWS environment using Java

Question

I would like to go through ALL generated logs on my AWS environment using Java API and look for a specific String on a given timeframe. I know how to do this with a specific log group as the following example shows but I would like to know if it is possible to iterate through all log groups without calling each by its name. Ideally I would have a forEach loop on all log events or at least be able to get all log streams and then run forEach on all of them.

Here is how I fetch one log group:

DescribeLogStreamsRequest describeLogStreamsRequest = new 
DescribeLogStreamsRequest().withLogGroupName("log-group-name");
DescribeLogStreamsResult describeLogStreamsResult = 
logsClient.describeLogStreams(describeLogStreamsRequest);

    for (LogStream logStream : describeLogStreamsResult.getLogStreams())
    {
        GetLogEventsRequest getLogEventsRequest = new GetLogEventsRequest()
                .withStartTime(1111100000L)
                .withEndTime(22222800000L)
                .withLogGroupName("log-group-name")
                .withLogStreamName(logStream.getLogStreamName() );

        GetLogEventsResult logEventResult = logsClient.getLogEvents(getLogEventsRequest);

        logEventResult.getEvents().forEach( outputLogEvent -> {
            System.out.println(outputLogEvent.getMessage());
        } );

    }

Answer 1

EDIT: THIS IS NOT A GOOD SOLUTION. So after actually running this code, I found out that each query is limited to 1000 results and I couldnt find any way to iterate or fetch more results from the same log group. END OF EDIT

So I eventually went through the following solution. I am fetching all log groups and run a log insight query on each. Here is the code:

public List<LogGroup> getAllLogGroups(String region) {
    List<LogGroup> logGroups = new ArrayList<>();
    logger.info(String.format("Getting all log groups from %s",region));
    try {
        ClientConfiguration clientConfig = new ClientConfiguration();
        AWSLogsClientBuilder awsLogsClientbuilder = AWSLogsClientBuilder.standard();
        AWSLogs logsClient = awsLogsClientbuilder.withRegion(region).withClientConfiguration(clientConfig).build();
        DescribeLogGroupsRequest describeLogGroupsRequest = new DescribeLogGroupsRequest();
        String nextToken = null;
        do {
            describeLogGroupsRequest.setNextToken(nextToken);
            DescribeLogGroupsResult response = logsClient.describeLogGroups(describeLogGroupsRequest);
            logGroups.addAll(response.getLogGroups());
            nextToken = response.getNextToken();
        } while (null != nextToken);
    } catch (Exception e) {
        logger.error(String.format("ERROR: Unable to get logger groups on %s - %s",region, e));
        throw e;
    }
    return logGroups;
}


public int getNumberOfOccurrencesFromLogGroups(List<LogGroup> logGroups, long startTime, long endTime, String searchedString) {

    AWSLogsClientBuilder awsLogsClientbuilder = AWSLogsClientBuilder.standard();
    AWSLogs logsClient = awsLogsClientbuilder.build();
    List<String> logs = new ArrayList<>();
    try {
        for (LogGroup logGroup : logGroups) {
            String logGroupName = logGroup.getLogGroupName();
            StartQueryRequest startQueryRequest = new StartQueryRequest();
            
            String query = "fields @message " +
                    "| filter @message like /"+searchedString+"/";

            startQueryRequest.setLogGroupName(logGroupName);
            startQueryRequest.setStartTime(startTime);
            startQueryRequest.setEndTime(endTime);
            startQueryRequest.setQueryString(query);
            ...