How to calculate my applications iops utilization

Question

I'm trying to figure out how I determine the IOPS my application is driving so I can property size our cloud infrastructure components. I understand what IOPS are between a database and the storage layer but I'd like to understand how I go about calculating what my application drives. Here are some of my applications characteristics: 1) 90% write and 10% read 2) We have a java based application that ultimately inserts into an HBase database 3) Process about 50 msg/sec where each message results in probably 2 HBase inserts

Here is what I'm not sure about: 1) Is the only way to calculate the IOPS is by running iostat or something on the actual server during load? 2) Is there a general way I can calculate what needed from the data volume/size coming in and not on the actual storage unit? 3) Is there any relationship to the # of transactions and the # of bytes in each transaction (I read somewhere an IO is usually 3K, most inserts don't contain that much info so it doesn't matter).

Any help would be greatly appreciated.

Answer 1

Not very familiar with Hbase. But from the documentation, it uses a log structure, which means the writes will be sequential writes. It also has compactions, which will cause both sequential reads and writes of multi-MB. The read queries will cause random reads on the storage layer. So here is the answer to your questions:

1. As far as I know, yes. The only way to get IOPS is running iostat. You can probably get some compaction stats from the application level. But it is hard to extract IOPS level details.

2. Compaction will cause more storage than the entire data size. And if your application is write heavy(compaction might not catch up with the speed of inserts), the size of actual data volume will be much larger. Given the 50 msg/sec in your question, this should not be the case. I will provision disks double the size of expected data volume per instance.

3. As mentioned above, Hbase is log structured. Writes are accumulated in memory and flushed to disk together. So it doesn't matter the size of each transaction.