Does running many IO threads impact performance of a few CPU-intensive threads?

Question

Let's say I have a 6-core machine with 12MB cache. I use it for a server application that has a few gigabytes of heap (much of it 2nd level Hibernate cache).

I noticed that most of the time I have a handful threads actively serving client requests (burning CPU and talking to DB), as well as about 30-50 more threads that are only doing ol' good synchronous network IO with client.

As I am learning about Java memory model, I am wondering if this can impact performance. Does context switching for one of the many network IO threads ruin thread/CPU cache of the "active" threads? Is this level of concurrency harmful in itself (memory cache aside)?

Does it really matter, given how small the CPU cache is in relation to the whole application memory? How can I determine where the boundary is?

Answer 1

Does context switching for one of the many network IO threads ruin thread/CPU cache of the "active" threads?

There are no hard guarantees, but my gut feeling is that, in practice, CPU caches get updated by order(s) of magnitude more frequently than the scheduler switches threads. If this is the case, then the cache update required by thread context switching is extremely small overhead compared to the cache updates that are going on already.

How can I determine where the boundary is?

By experimenting and measuring (profiling), as is the case with most performance-related questions. Simple theoretical approximations won't work because the reality of what's really going on in a modern PC is more complex than any reasonable approximation.

Answer 2

There will be an impact and you can get some jitter when you have more active threads than cpus. However, the jitter is usually considered to be relatively small 10 to 100 micro-seconds.

You can have the thread competing for cache resources, but there is not much you can do about it except reduce the amount of active data each thread uses. This can reduce your cpu load by a factor of 2x or more, but involves alot of work, possibly a complete re-write (including some JDK and third party libraries) and it may yeild little benefit. eg how much do you need the cpu load to be lower.