Failed to connect to Tomcat server on ec2 instance

Question

UPDATE :

1. My goal is to learn what factors could overwhelm my little tomcat server. And when some exception happens, what I could do to resolve or remediate it without switching my server to a better machine. This is not a real app in a production environment but just my own experiment (Besides some changes on the server-side, I may also do something on my client-side)

2. Both of my client and server are very simple: the server only checks the URL format and send 201 code if it is correct. Each request sent from my client only includes an easy JSON body. There is no database involved. The two machines (t2-micro) only run client and server respectively.

3. My client is OkHttpClient() . To avoid timeout exceptions, I already set timeout 1,000,000 milli secs via setConnectTimeout , setReadTimeout , and setWriteTimeout . I also go to $CATALINA/conf/server.xml on my server and set connectionTimeout = "-1" (infinite)

ORIGINAL POST:

I'm trying to stress out my server by having a client launching 3000+ threads sending HTTP requests to my server. Both of my client and server reside on different ec2 instances.

Initially, I encountered some timeout issues, but after I set the connection, read and write timeout to a bigger value, this exception has been resolved. However, with the same specification, I'm getting java.net.ConnectException: Failed to connect to my_host_ip:8080 exception. And I do not know its root cause. I'm new to multithreading and distributed system, can anyone please give me some insights of this exception?

Below is some screenshot of from my ec2:

1. Client:

2. Server:

Answer 1

Having gone through similar exercise in past I can say that there is no definitive answer to the problem of scaling.

Here are some general trouble shooting steps that may lead to more specific information. I would suggest trying out tests by tweaking a few parameters in each test and measure the changes in Cpu, logs etc.

1. Please provide what value you have put for the timeout. Increasing timeout could cause your server (or client) to run out of threads quickly (cause each thread can process for longer). Question the need for increasing timeout. Is there any processing that slows your server?

2. Check application logs, JVM usage, memory usage on the client and Server. There will be some hints there.

3. Your client seems to be hitting 99%+ and then come down. This implies that there could be a problem at the client side in that it maxes out during the test. Your might want to resize your client to be able to do more.

4. Look at open file handles. The number should be sufficiently high.

5. Tomcat has some limit on thread count to handle load. You can check this in server.xml and if required change it to handle more. Although cpu doesn't actually max out on server side so unlikely that this is the problem.

6. If you a database then check the performance of the database. Also check jdbc connect settings. There is thread and timeout config at jdbc level as well.

7. Is response compression set up on the Tomcat? It will give much better throughout on server especially if the data being sent back by each request is more than a few kbs.

--------Update----------

Based on update on question few more thoughts.

Since the application is fairly simple, the path in terms of stressing the server should be to start low and increase load in increments whilst monitoring various things (cpu, memory, JVM usage, file handle count, network i/o).

The increments of load should be spread over several runs. Start with something as low as 100 parallel threads.

Record as much information as you can after each run and if the server holds up well, increase load.

Suggested increments 100, 200, 500, 1000, 1500, 2000, 2500, 3000.

At some level you will see that the server can no longer take it. That would be your breaking point.

As you increase load and monitor you will likely discover patterns that suggest tuning of specific parameters. Each tuning attempt should then be tested again the same level of multi threading. The improvement of available will be obvious from the monitoring.