Performance of instance store vs EBS-optimized EC2 or RAID volumes

Question

As far as I can tell from my own experience and from what I have read, there are very few situations in which one wouldn't want to use EBS over instance store. However, instance store is generally faster for disk read/writes due to it's being physically attached to the EC2. How much faster, and whether it is faster in all cases, I don't know.

So, what I am curious about, if anyone out there has had some experience with any of these, is the relative speed/performance of:

1. An EC2 using instance store vs a non-storage-optimized EC2 using EBS (of any storage type)

2. An EC2 using instance store vs a storage-optimized (I3) EC2 using EBS

3. An EC2 using instance store vs a non-storage-optimized EC2 using some kind of EBS RAIDing

4. A non-storage-optimized EBS-backed EC2 vs a storage-optimized EC2 vs an EC2 with an EBS RAID configuration

5. All of the above vs EBS-optimized instances of any type.

The more specific and quantifiable the answers the better -- thanks!

Answer 1

Now Available – I3 Instances for Demanding, I/O Intensive Applications claims that Instance Store on i3 instances:

can deliver up to 3.3 million IOPS at a 4 KB block and up to 16 GB/second of sequential disk throughput.

Coming Soon – The I2 Instance Type – High I/O Performance Via SSD claims that Instance Store on i2 instances:

deliver 350,000 random read IOPS and 320,000 random write IOPS .

Amazon EBS Volume Types lists:

• General Purpose SSD: Maximum 10,000 IOPS/Volume
• Provisioned IOPS SSD: Maximum 20,000 IOPS/Volume
• Throughput Optimized HDD: Maximum throughput 500 MiB/s (Optimized for throughput rather than IOPS, good for large, contiguous reads)

Answer 2

We did a thorough set of benchmarks for some of those situations comparing EBS vs. instance store, namely,

1. EBS SSD general iops (about 3k for a 1TB volume)
2. EBS SSD provisioned iops (about 50k for a 1TB volume)
3. instance store local disk (one disk raid 0)
4. instance store local disk (two disks striped raid 1)

We had the following takeaways,

1. the local disk options are vastly better at random access than EBS, due to low latency and iops being a limiting factor. Although most applications try to avoid random reading, it's hard to avoid completely and so good performance in this area is a big plus.

2. Sequential reads are also vastly better than EBS, mainly due to rate limiting of EBS, specifically the throughput. Generally you are going to get full, unrestricted access to a local disk with much lower latency than network storage (EBS).

3. Raid1 is (not surprisingly) up to 2x better for reads than the single disk. Writes are the same due to needing to write to both disks. However on larger system, you can have 4+ disks and do raid10 (mirrored striping) which would give improvements to writes as well.

4. Unfortunately as mentioned at the start, local disk options are ephemeral and will lose your data on a terminate/stop of the instance. Even so, it might be worth considering a high availability architecture to allow using it.

5. EBS 50K is certainly more performant than 3K, although you generally need to get past 4+ threads to see a real difference (eg a database). Single threaded processes are not going to be much faster (eg a file copy, zip, etc..). EBS 50k was limited by the instance max iops (30k), so generally be aware the instance size also can be a limiting factor on EBS performance.

6. It's possible to raid EBS as well, but keep in mind it's networked storage and so that will likely be a real bottleneck on any performance gains. Worth a separate test to compare.

full details on the benchmarks can be found at,

https://www.scalebench.com/blog/index.php/2020/06/03/aws-ebs-vs-aws-instance-storelocal-disk-storage/