New AWS EC2 EBS gp3 volumes are slow

Question

Running sysbench against volumes I'm finding the gp3 volumes are much slower. Even when I provision a volume with 16000IOPS after waiting for optimization I'm getting a cap at 1000 IOPS when monitoring through Percona PMM, New Relic CloudWatch.

Any ideas? Thanks, Jeremy

Instance:

• C5.4xlarge Ubuntu 18.04

Volumes:

• gp2 3000GB (gives 9000 IOPS)
• gp3 3000GB 9000 IOPS 250/s

Sysbench results below:

sysbench --file-total-size=15G --file-num=16 fileio prepare
sysbench --file-total-size=15G --file-num=16 --file-test-mode=rndrw --time=600 fileio run
sysbench --file-total-size=15G --file-num=16 fileio cleanup

gp3 | 9000 | 3000 | 250/s

File operations:
reads/s: 576.37
writes/s: 384.24
fsyncs/s: 153.70

Throughput:
read, MiB/s: 9.01
written, MiB/s: 6.00

General statistics:
total time: 600.0333s
total number of events: 668612

Latency (ms):
min: 0.00
avg: 0.90
max: 337.40
95th percentile: 3.89
sum: 599693.33

Threads fairness:
events (avg/stddev): 668612.0000/0.00
execution time (avg/stddev): 599.6933/0.00

gp2 | 9000 | 3000 | gp2

File operations:
reads/s: 1523.68
writes/s: 1015.79
fsyncs/s: 406.33

Throughput:
read, MiB/s: 23.81
written, MiB/s: 15.87

General statistics:
total time: 600.0064s
total number of events: 1767487

Latency (ms):
min: 0.00
avg: 0.34
max: 70.10
95th percentile: 1.06
sum: 599390.12

Threads fairness:
events (avg/stddev): 1767487.0000/0.00
execution time (avg/stddev): 599.3901/0.00

Percona PMM gp3 > gp2 comparison

Answer 1

Anyone have ideas on this? I've posted on the AWS forums as well with no response.

Here are some more test results with the same volumes as above except I increased the gp3 volumes to 16000 IOPS and 1000/s throughput and still get worse results.

• Sequential writes 2x performance on gp2s
• Sequential reads = performance
• Random writes = performance
• Random reads 2x performance on gp2s
• Sequential read/writes +50% performance on gp2s

Pics are 2 gp3 volumes on the left and the old gp2 on the right. If there are two humps the first one is writing the sysbench test files, second is the test.

Sequential Writes

Sequential Reads

Random Writes

Random Reads

Sequential Read/Writes

Thanks, Jeremy

Answer 2

My initial enthusiasm for gp3 was dampened by inferior performance to gp2, however, when I set the size, IOPS, and bandwidth the same on C5, M5, and M5A instance type I got similar performance measurements from fio : https://github.com/axboe/fio

I'm testing on CentOS 7.8, fio-3.7-2.el7.x86_64 . Region us-east-1 .

Are you sure sysbench is the right benchmarking tool? The fileio options don't seem greatly documented, and I can get wildly different results when changing the sync mode, flags, etc.

Your example command equivalent to this because of the default values:

sysbench --file-total-size=15G --file-num=16 --file-test-mode=rndrw --file-io-mode=sync --file-rw-ratio=1.5 --file-fsync-freq=100 --max-requests=0 --time=600 fileio run

When I run a different sysbench command, notably using direct access and fsync=off :

sysbench fileio --file-total-size=15G --file-test-mode=rndrw --time=600 --max-requests=0 --file-io-mode=async --file-extra-flags=direct  --file-num=16 --file-rw-ratio=1 --file-fsync-freq=0 run

I get equivalent performance from gp2 and gp3. (Actually, gp3 is slightly better).

Here's a summary of my fio data, using the command

fio --rw=TEST --direct=1 --ioengine=libaio --bs=16k --numjobs=8 --size=1G --group_reporting

with TEST= {read,write,randread,randwrite,randrw}

gp2:

gp2-randread-16k.out:   read:   IOPS=3081,  BW=48.2MiB/s  (50.5MB/s)(8192MiB/170117msec)
gp2-randrw-16k.out:     read:   IOPS=1540,  BW=24.1MiB/s  (25.2MB/s)(4095MiB/170121msec)
gp2-randrw-16k.out:     write:  IOPS=1541,  BW=24.1MiB/s  (25.2MB/s)(4097MiB/170121msec)
gp2-randwrite-16k.out:  write:  IOPS=3081,  BW=48.2MiB/s  (50.5MB/s)(8192MiB/170116msec)
gp2-read-16k.out:       read:   IOPS=3090,  BW=48.3MiB/s  (50.6MB/s)(8192MiB/169642msec)
gp2-write-16k.out:      write:  IOPS=3081,  BW=48.2MiB/s  (50.5MB/s)(8192MiB/170115msec)

gp3:

gp3-randread-16k.out:   read:   IOPS=3080,  BW=48.1MiB/s  (50.5MB/s)(8192MiB/170188msec)
gp3-randrw-16k.out:     read:   IOPS=1539,  BW=24.1MiB/s  (25.2MB/s)(4095MiB/170238msec)
gp3-randrw-16k.out:     write:  IOPS=1540,  BW=24.1MiB/s  (25.2MB/s)(4097MiB/170238msec)
gp3-randwrite-16k.out:  write:  IOPS=3082,  BW=48.2MiB/s  (50.5MB/s)(8192MiB/170073msec)
gp3-read-16k.out:       read:   IOPS=3089,  BW=48.3MiB/s  (50.6MB/s)(8192MiB/169719msec)
gp3-write-16k.out:      write:  IOPS=3083,  BW=48.2MiB/s  (50.5MB/s)(8192MiB/170053msec)