Elasticsearch: high CPU usage

Question

I'm running ES on an AWS EC2 t2.small instance, and I am experiencing a sudden and massive drop in CPU credits every once in a while.

https://www.dropbox.com/s/0pw0qfudoca899f/cpu_credits.png?dl=0

The drop started on a monday, which is when we create 4 new logging indices for that weeks logs. We currently have ~60 logging indices which mostly just get insert requests and rarely any searching is done. We also have about 30 indices that are actively searched against and at least 10 of them get regular bulk updates.

Last time I was faced with an issue like this, I deleted a bunch of old indices and that seemed to help, however I would prefer avoiding that.

What are the most common reasons for high resource usage? Amount of indices? Amount of records in them? Amount of shards allocated? Amount of updates to records or mapping (there are a some indices with thousands of fields)?

Let me know if there's any information I could provide, and thank you in advance for any help on clearing this issue.

---

EDIT 1:

Output from _cat/indices?v

Output from _nodes/stats

Answer 1

So with a t2.small I suppose you have 1GB of RAM allocated to the ES heap, right? One thing that I notice is that given the very small size of your indices (< 100mb), you have way too many shards, a single shards would be more than enough. Since each shard consumes resources, you'd be way better off.

One thing you can do is to consolidate all your indices, ie put all go_request_data-2016 weekly indices into a yearly one with a single shard, etc. You'd probably end up with way many indices and shards without having to delete any data