AWS Lambda (Java) invoking wrong instance of Object

Question

I have a unusual problem with AWS Lambda invocations, did not find much about this when searching. I'm hoping someone here has a deeper understanding of Java garbage collector and/or AWS lambda containers.

I've created a custom PrintStream to catch printouts from System.out and System.err. The purpose is to log these printouts through the logger to add useful some more useful information to the messages.

CustomPrintStream.write is triggered on every invocation after the first one it still uses the object from the first invocation. Which means that when the variable timeWhenFunctionStarted i used it contains the value from the first invokation and the time elapsed is incorrect.

I've added printout of the objects hashcode to better illustrate this problem.

Here are the logs

During the first invocation (cold start) everything is as expected, consistent ms and same hashcode:

START RequestId: {requestid1} Version: $LATEST
2019-01-09 12:37:09.175 [INFO]  {requestid1} START  (1 ms)      hashcode: 601008104 //other stuff
2019-01-09 12:37:09.193 [DEBUG] {requestid1}        (18 ms)     hashcode: 601008104 //other stuff
2019-01-09 12:37:09.213 [INFO]  {requestid1}        (38 ms)     hashcode: 601008104 //other stuff
2019-01-09 12:37:13.813 [TRACE] {requestid1}        (4638 ms)   hashcode: 601008104 //other stuff
2019-01-09 12:37:16.143 [INFO]  {requestid1}        (6968 ms)   hashcode: 601008104 //other stuff
2019-01-09 12:37:16.143 [INFO]  {requestid1} END    (6968 ms)   hashcode: 601008104 //other stuff
END RequestId: {requestid1}
REPORT RequestId: {requestid1} Duration: 7207.15 ms Billed Duration: 7300 ms Memory Size: 512 MB Max Memory Used: 113 MB

The issue occurs when the lambda function is invoked again in the same container, see hashcode, requestid and ms for trace logging event:

START RequestId: {requestid2} Version: $LATEST
2019-01-09 12:37:29.717 [INFO]  {requestid2} START  (0 ms)      hashcode: 2117173674 //other stuff
2019-01-09 12:37:29.717 [DEBUG] {requestid2}        (1 ms)      hashcode: 2117173674 //other stuff
2019-01-09 12:37:29.718 [INFO]  {requestid2}        (1 ms)      hashcode: 2117173674 //other stuff
2019-01-09 12:37:29.815 [TRACE] {requestid1}        (20640 ms)  hashcode: 601008104  //other stuff
2019-01-09 12:37:30.075 [INFO]  {requestid2}        (358 ms)    hashcode: 2117173674 //other stuff
2019-01-09 12:37:30.075 [INFO]  {requestid2} END    (358 ms)    hashcode: 2117173674 //other stuff
END RequestId: {requestid2}
REPORT RequestId: {requestid2} Duration: 358.78 ms Billed Duration: 400 ms Memory Size: 512 MB Max Memory Used: 116 MB 

Logging class:

public class CustomLogger {

    //Constructor setting System out to object of class customPrintStream
    public CustomLogger(...) {
        //other stuff
        this.logger = context.getLogger();
        this.baosSystemOut = new ByteArrayOutputStream();
        this.customPrintStreamObject= new customPrintStream(baosSystemOut);
        System.setOut(customPrintStreamObject);
    }

    //Every logged message goes through here.
    private void logMessages(String[] messages, String logLevel) {
        //other stuff
        String formatedMessage = ... System.currentTimeMillis() - timeWhenFunctionStarted ... System.hashcode(this) ...;
        LOGGER.log(formatedMessage);
    }

    //called from CustomPrintStream class and from Lambda function handler, same for TRACE, DEBUG, INFO, WARN and ERROR.
    public class logTRACE(String... messages){
        //other stuff
        logMessages(messages, "TRACE");
    }

    //private class to catch logging
    private class CustomPrintStream extends PrintStream {
        //other stuff
        @Override
        public void write(byte[] buf, int off, int len) {
            //other stuff
            logTRACE(message);
        }
    }
}

CustomLogger is called from Lambda function handler where issue is not present:

//other stuff
CustomLogger logger = new CustomLogger(...);
logger.logINFO(...);

The problem occurs when logging is called from CustomPrintStream class, I've tried setting CustomPrintStream and the logger to null after program completion and manually calling the finalize method from the super class but that does not change anything.

What i want to happen is for the printouts to use the CustomPrintStream in the logger object for the current invocation.

I'm grateful for answers.

Answer 1

It sounds like you are creating CustomPrintStream in a static context. Due to the way container reuse works in AWS Lambda, your statically defined objects will persist across invocations.

Answer 2

Solved the issue, kind of at least. Moved CustomPrintStream out of CustomLogger class.