How to get the Aws cloudwatch logs using java

Question

We are working on AWS Lambda and Cloudwatch logs. Now, I want fetch all the log events from the Cloudwatch logs without using logStreamName by Java.

Since we are generating the logstream in dynamic way, am not sure how to fetch all the logs from the Cloudwatch log group.

I know, If we have logstream name, then we can use the below code

ClientConfiguration clientConfig = getClientConfig();

AWSLogsClientBuilder builder = AWSLogsClientBuilder.standard();

AWSLogs logsClient= builder.withCredentials(new AWSStaticCredentialsProvider(new ProfileCredentialsProvider(profile).getCredentials())).withRegion(Regions.AP_SOUTHEAST_2).withClientConfiguration(clientConfig).build();

GetLogEventsRequest request = new GetLogEventsRequest()
                        .withStartTime(1531231200000L)
                        .withEndTime(1531576800000L)
                        .withLogGroupName("FlowLogs_GroupName")
                        .withLogStreamName("eni-xxxxx");

GetLogEventsResult result = logsClient.getLogEvents(request);

result.getEvents().forEach(outputLogEvent -> {
          System.out.println(outputLogEvent.getMessage());
}); 

Since am new to this AWS, can anyone please help me with some code samples?

Answer 1

You can use DescribeLogStreamsRequest to get log stream name. Hope this would help

  public static void main( String[] args )
        {
            ClientConfiguration clientConfig = new ClientConfiguration();

            AWSLogsClientBuilder builder = AWSLogsClientBuilder.standard();

            AWSLogs logsClient = builder.withCredentials( new AWSStaticCredentialsProvider( new ProfileCredentialsProvider().getCredentials() ) )
                    .withRegion( Regions.AP_SOUTHEAST_2 )
                    .withClientConfiguration( clientConfig ).build();

            DescribeLogStreamsRequest describeLogStreamsRequest = new DescribeLogStreamsRequest().withLogGroupName( "FlowLogs_GroupName"  );
            DescribeLogStreamsResult describeLogStreamsResult = logsClient.describeLogStreams( describeLogStreamsRequest );

            for ( LogStream logStream : describeLogStreamsResult.getLogStreams() )
            {
                GetLogEventsRequest getLogEventsRequest = new GetLogEventsRequest()
                        .withStartTime( 1531231200000L )
                        .withEndTime( 1531576800000L )
                        .withLogGroupName( "FlowLogs_GroupName" )
                        .withLogStreamName( logStream.getLogStreamName() );

                GetLogEventsResult result = logsClient.getLogEvents( getLogEventsRequest );

                result.getEvents().forEach( outputLogEvent -> {
                    System.out.println( outputLogEvent.getMessage() );
                } );

            }
        }

Answer 2

All the log stream prefix/suffix are dynamically generated and stored in ECS Cluster->Service->task You can use the below piece of code to obtain the prefix/suffix and later use this in your log stream name

AmazonECS ECSclient = AmazonECSClientBuilder.standard().withClientConfiguration(clientConfig).withRegion(Regions.AP_SOUTHEAST_2).build();
        ListTasksRequest request = new ListTasksRequest().withCluster("YourClusterName").withServiceName("YourServiceName");
        ListTasksResult response = ECSclient.listTasks(request);
String taskSuffix = response.getTaskArns().get(0)

Use this taskSuffix in the logStream Name

Hope this helps.

Answer 3

You can create resources/log4j2.xml file in the main package and create the content of sl4j2.xml as follows the link. https://docs.aws.amazon.com/lambda/latest/dg/java-logging.html Then use the logger object as follows.

private static final Logger logger = LoggerFactory.getLogger(Handler.class);

Or else use an object of LambdaLogger as following the code you can see.

LambdaLogger logger = context.getLogger();
logger.log("This should be log")

Answer 4

In addition to https://stackoverflow.com/a/54074465/11090297

If you need filtered logs Stream data then you can use FilterLogEventsRequest

Here is an example of it:

 {
    ClientConfiguration clientConfig = new ClientConfiguration();


  AWSLogsClientBuilder builder = AWSLogsClientBuilder.standard();

    AWSLogs logsClient = builder.withCredentials((new ClasspathPropertiesFileCredentialsProvider("aws.properties")))
        .withRegion(Regions.US_EAST_1)
        .withClientConfiguration(clientConfig).build();

    DescribeLogStreamsRequest describeLogStreamsRequest = new DescribeLogStreamsRequest().withLogGroupName("/aws/audit");
    DescribeLogStreamsResult describeLogStreamsResult = logsClient.describeLogStreams(describeLogStreamsRequest);

    for (LogStream logStream : describeLogStreamsResult.getLogStreams()) {

// Add FilterPattern which will only fetch logs required
      FilterLogEventsRequest filterLogEventsRequest =  new FilterLogEventsRequest().withLogGroupName("/aws/audit")
         .withLogStreamNames(Arrays.asList(logStream.getLogStreamName())).withFilterPattern("vayuj");
      FilterLogEventsResult result = logsClient.filterLogEvents(filterLogEventsRequest);

      result.getEvents().forEach(outputLogEvent -> {
        System.out.println(outputLogEvent.getMessage());
      });

    }
  }