调整的adm更改，吞吐量从4k下降到9条消息

Question

I have a network client and server application. The dataflow is such that the client sends a message to the server and the server responds with an acknowledgment. Only on the receipt of the acknowledgment, client seconds the next message.

The client application, written in C++, have 3 threads, namely network thread (responsible for sending messages via socket), main thread( responsible for making a request message) and a timer thread (fires every second).

The server application have 2 threads, main thread and the network thread.

I run RHEL 6.3, 2.6.32-279 kernel.

Configuration 1

1. tuned-adm profile latency-performance
2. All Client's threads on same CPU Core id
3. All Server's threads on same CPU Core id, but a different Core Id from Client's thread
4. Client and server running on same machine

Throughput: 4500 messages per second

Configuration 2

1. tuned-adm profile throughput-performance
2. All Client's threads on same CPU Core id
3. All Server's threads on same CPU Core id, but a different Core Id from Client's thread
4. Client and server running on same machine

Throughput: 9-15 messages per second

Configuration 3

1. tuned-adm profile throughput-performance
2. All Client's threads on different CPU Core id
3. All Server's threads on different CPU Core id, and different Core Id from Client's thread
4. Client and server running on same machine

Throughput: 1100 messages per second

The machine has negligible load. Can someone explain the drop from 4k to 9 messages per second when profile was switched from latency-performance to throughput-performance.

Answer 1

Here's the basic schedule of differences between the RHEL tuned-adm profiles:

Latency performance shifts the I/O elevator to deadline and changes the CPU governor to the "performance" setting.

Throughput performance is optimized for network and disk performance. See the specifics below...

Your workload appears to be latency sensitive.

Here's the setup for throughput-performance w/ comments. latency-performance does not modify any of these.

# ktune sysctl settings for rhel6 servers, maximizing i/o throughput
#
# Minimal preemption granularity for CPU-bound tasks:
# (default: 1 msec#  (1 + ilog(ncpus)), units: nanoseconds)
kernel.sched_min_granularity_ns = 10000000

# SCHED_OTHER wake-up granularity.
# (default: 1 msec#  (1 + ilog(ncpus)), units: nanoseconds)
#
# This option delays the preemption effects of decoupled workloads
# and reduces their over-scheduling. Synchronous workloads will still
# have immediate wakeup/sleep latencies.
kernel.sched_wakeup_granularity_ns = 15000000

# If a workload mostly uses anonymous memory and it hits this limit, the entire
# working set is buffered for I/O, and any more write buffering would require
# swapping, so it's time to throttle writes until I/O can catch up.  Workloads
# that mostly use file mappings may be able to use even higher values.
#
vm.dirty_ratio = 40