AWS http api gateway + lambda (node/express) Internal Server Error

Question

. 。 Answers to this question are eligible for a +100 reputation bounty. micronyks is looking for a to this question: ：

I need proper direction and way to fix this. I would really appreciate the help. This is really critical and I don't know AWS service much.

I get internal server error when I have a long running query. Actually, I have to fetch historic data through an API, which sometime can take longer than 30 seconds. It depends on the query how complex it is. It can take 1 min also.

Not sure but guessing, API gateway timeout is set to 30 seconds (and I cann't increase it) and my query execution time is more then 30 seconds. So I get internal server error I believe.

HOW can I say above statement?

because If I run the same query locally, I mean in node/express locally by running npm run start , it works fine even if takes1 mins, response will always come back.

But when I deploy node/express code to lambda function, it throws error if any query takes longer period to execute.

I have following setup of node/express

const express = require("express");
const serverless = require("serverless-http");
const app = express();

app.use(cors());
app.use((req, res, next) => {
    res.setHeader('Connection', 'keep-alive');                   // I added this line as suggested in some post but not helping
    res.setHeader('Keep-Alive', 'timeout=30');                   // I added this line as suggested in some post but not helping
    res.setHeader("Access-Control-Allow-Headers", "X-Requested-With,content-type");
    res.setHeader("Access-Control-Allow-Origin", "*");
    res.setHeader("Access-Control-Allow-Methods", "GET, POST, OPTIONS, PUT, PATCH, DELETE");
    res.setHeader("Access-Control-Allow-Credentials", true);
    next();
});

app.use(express.json());
app.use(express.urlencoded({ extended: true }));

app.use(`api-end-point/user`, userRoute);
....

if (process.env.NODE_ENV !== "lambda") {

    PORT = process.env.PORT || 7000;
    const server = app.listen(PORT, () => {
        console.log(`node-express server running in ${process.env.NODE_ENV} mode on ${PORT}`);
    });
    server.timeout = 0;

}else {

    module.exports.handler = serverless(app);     // this is for lambda function

}

I deploy this code to AWS lambda function.

---

HTTP API gateway is configured with two routes /ANY, /{proxy+}

---

TIMEOUT

API gateway is set to default 30 seconds . [I can not increase this time as not allowed by AWS]

Lambda is set to 10 **mins**

---

CORS

---

I really have no idea how can I fix this problem ?

How can I increase API gateway timeout or How can I keep connection alive?

Answer 1

https://docs.aws.amazon.com/apigateway/latest/developerguide/limits.html

Since timeout cannot be incresed, you might change using single HTTP request to

• https://en.wikipedia.org/wiki/Post/Redirect/Get Pattern

1. Client POST query
2. Server response an url for the result
3. Client GET the url multiple times -- it will be 200 OK when the result ready

• or WebSocket

The document says Idle Connection Timeout for WebSocket is up to 10 minutes

Answer 2

You cannot increase the API Gateway timeout to greater than 30 seconds, as has already been mentioned.

The only solution I know of at this time is to run your Lambda asynchronously, but this cannot be done in an Http API. But if you're willing to change it to a REST API, then this can be done with a combination of turning on Lambda Proxy Integration in the REST API and invoking the Lambda asynchronously utilizing an invoke header X-Amz-Invocation-Type. This will allow your Lambda to run asynchronously (up to 15 minutes) with an API call.