How to setup HTTP server with nodejs for logging?

Question

I need to setup a logging service for my startup. I and my team do not have the time and resources (monetary) to deploy a dedicated tool for logs. However I need to run analytics on user behavior. While I am going to learn the analytics bit and implement it down the line, I need to log the user activity so that I can pull out the data later and do the math.

I am using a SQL based (RDBMS) setup for my main application (which is a web service) and plan to log user activity in DynamoDB (AWS).

Also, I plan to setup a local (on the same server) Node.Js server which can handle log requests coming from my PHP based app to make sure that the logging activity runs in another process and is handled gracefully without crashing or slowing down the main server code. The idea here is - when a request comes in from the user, the PHP application would make call to the NodeJS server. The NodeJS server will take the request and return immediately while queuing the request (like say 200 other requests) and the PHP app will continue to serve the request. The NodeJS server will go on logging everything in the queue one after the other.

The reason why I plan to do this is - NodeJS runs on a single thread and is event-loop based - hence, in case there is a sudden traffic spike, log requests would be queued and the data will be sent to Dynamo eventually without making the PHP application wait for the confirmation.

Note : When I say user activity , I mean application-level logging like:

• User X from ABCD IP address tried to access an article with ID 123 at 2016-09-10 14:21:22
• User Y Posted Comment #234 on Article #987 at 2016-09-10 18:30:02 using ABC app.
• User C logged in to the web interface from EFGH IP address.

Of course, the data would be in JSON format and the server will make calls to the AWS SDK.

Here, the NodeJs server would basically be acting as a log-proxy. Based on the idea/scenario so far, I have the following questions:

1. Is the decision to use DynamoDB for logging requests okay or am I making a mistake? The service fits within my budget.
2. What kind of limitations will I have with NodeJS in this scenario? Is nodejs okay for it?
3. What Node modules should I use for this? Will HTTP server do or are there modules openly available which are made for such scenarios?
4. How to I make sure that this server listens only to requests coming from the same machine (loopback)?

Answer 1

Here will go an TL;DR; answer:

1 - Is the decision to use DynamoDB for logging requests okay or am I making a mistake? The service fits within my budget.

DynamoDB is an nosql database service that provides fast and predictable performance with seamless scalability, IMO it's a good way to use when we talk about loggin, as expected you need a fast writing and fast reading, without relationship between logs messages (I think). So you can use DynamoDB without problems here.

2 - What kind of limitations will I have with NodeJS in this scenario? Is nodejs okay for it?

NodeJS is (as you said) single thread and is event-loop based, if you have more requests than NodeJS can handle with, you should fork or something like

I still think that you should use a tool like Logstash but if you sure that you can use NodeJS to do that, probably you will have success.

3 - What Node modules should I use for this? Will HTTP server do or are there modules openly available which are made for such scenarios?

When I talk about logging strategies, the first thing that coming in my mind is "thousands of packages over the network", I don't know if you have this in your startup, anyway I suggest you to use UDP to send the logg messages, UDP (User Datagram Protocol) that is used primarily for establishing low-latency and loss tolerating connections between applications on the Internet. You can startup a UDP NodeJS listener using, for example dgram . To send the UDP log messages to your NodeJS server, you can use the Winston loggin library with the winston-udp plugin . Or, in your case, from PHP server, you can send UDP messages with socket or take a look here . Disclaimer: I'm not specialist in PHP, so I Googled and I found these solutions to send log messages over UDP, I'm not sure if this is the best way to do that :)

4 - How to I make sure that this server listens only to requests coming from the same machine (loopback)?

You can do this using a Security Group

Hope these help.