Performances spikes on Node.js application using LoopBack and Kerberos

Question

Our API's implemented with Node.js using the LoopBack framework retrieving data from HBase using Kerberos as authentication layer show inexplicable performance spikes of 5000 ms and 10000 ms with the bulk of the requests returning a result within 1000 ms.

performance spikes

Our applications are running as a Docker container on OpenShift. We are running Node.js version 6.13 and our OS is Red Hat Linux 7.3

We tried pinpointing the issue performing the following actions:

1. Run with and without Kerberos. Although running the application with Kerberos seems to worsen the spikes running the application without Kerberos shows more or less the same patterns. Therefore we ruled out Kerberos as the source of the problem.

2. Run the application locally instead of in OpenShift. The spikes occur both locally as in OpenShift. Therefore we ruled out OpenShift as the source of the problem.

3. Create the application with and without LoopBack. The spikes occur on both applications and therefore we also ruled out LoopBack as the source of the problem.

4. Using Node.js (with LoopBack) with an in-memory database. When using an in-memory database and not performing any external requests we don't see spikes anymore. From this we concluded that Node.js itself is not the problem but something seems to be going on using the Node.js external requests.

We are looking for suggestions what might cause our performance issues or where we could investigate further.

Answer 1

We have solved this issue, it did turn out to be the way NodeJS handles DNS lookup requests.

At first we did not believe this could be the case, no other application had connectivity issues and launching the same requests using curl resulted in no time-out's. After using systrace we noticed that one out of every 100 DNS lookup requests took an awful long time, coinciding with the spikes we saw earlier. After adding the IP addresses to the hosts file on each machine running our Node code the issue was resolved.