使用AWS Lambda进行长时间轮询

Question

I'm creating a monitoring system for an external API on which I need to invoke a function if the response is different from the initial request.

Currently without any long polling around my functions I've got the following:

  let gameAPI = new game(
      [
        "email@email.com",
        "password",
        "secret",
        "secret"
      ]
  );


  let lastModified;

  gameAPI.login().then(() => {

      gameAPI.getProfileStats(process.argv[2], process.argv[3], process.argv[4]).then(stats => {
          //first run grab this only
          if(lastModified < stats.data.lastModified) {
              //we found new data, stop polling & update mySQL database.
          } else {
              //continue to run every 1 minute...
          }
      });

  });

Would AWS Lambda be a good solution for this theory? if so, how would I go about doing it since Lambda charges for computer power usage?

Answer 1

No definitely not a good idea, in that case you are not letting the Lambda function exit and you are continuously billed. And on top of that Lambda has maximum execution time of 15min so you will need to somehow trigger it every so often.

One viable approach would be still to use Lambda, but instead of continue to run, exit on the else statement, and use CloudWatch trigger to invoke your function every minute, in that way you can run the check every minute, but only charges for the duration your checking function executes.

Edit

If there are many users to monitor, then depends on the actual number, a simple solution would be have an EC2 with a long living process, and you store Last Checked timestamp in DB. For each period (say 1min), select the DB for items last updated > 1min ago, and store in a queue within that local process. And in the process, at a certain rate (say 1/sec), do your fetch and check logic and update data and Last Checked in DB.

If you are expecting large volume, you can use SQS + Lambda w/CloudWatch + ELB, with Last Checked timestamp:

1. Every minute, CloudWatch invoke Lambda and select records that requires update and put the IDs into a SQS queue.
2. Use ELB worker environment and consume the queue to do the update.

That way you can have virtually infinitely rate. Or if you like slightly less complexity you can use simple EC2 instance and manually manage the instances and scale.

Answer 2

Will be good to have "GAME API" pushing stats to a SQS queue.

Then consumers will long poll the queue and save the results to wherever you want. This has several benefits.

1. You get long polling feature out of the box from SQS (No need to implement it)
2. By enabling long polling you reduce the number of requests to your game API thus you also reduce the costs
3. You can scale up and down your consumers whenever you want, you'll not run in risk of duplicates because SQS allows setting a visibility timeout for messages

Saving data into relations DB such as MySQL in your consumer code may lead to unexpected issues such as table locking when you have more than 1 consumer polling the same queue. Therefore it is a good Idea to use NoSQL DB for such solutions because they are flexible in terms of scalability.

For example, Dynamo DB has the ability to specify WCU (Write Capacity Units) for tables, thus you save costs and have control over what you provision.

Finally, SQS can be integrated with Lambda function (Async Lambda call), as a consumer, this ensures that you run the function only when it is needed because SQS triggers the function when there is a record to save.

So the answer to your question is yes it is a good idea to use Lambda, but you should use it with SQS queue.

I hope this helps.