Impact of frequent updates in elasticsearch

Question

We are planning to use elasticsearch to store merchant profile details and summary of his transaction amount. We have to update document for every transactions to maintain the total amount in sync.

1. Is it ok to update document frequently to increment a field?
2. We are planning to use update API to update a filed, Will it reindex the whole document? Is there any option to update only specific field without reindexing other fields?
3. It will be good if we have two index or a index with two types, one for profile other for summary?

Thanks in advance.

Answer 1

Please find the answers inline:

1. Is it ok to update documents frequently to increment a field?

It's very subjective and based on various parameters like how many documents in your index, how many shards, how many data nodes and what is your write QPS etc, Elasticsearch is very scalable and if you tune your cluster and index, you can get very high throughput without any issues. If you provide more details, we can suggest how to tweak this for a write-heavy system like yours. Please refer ES blog for tunning the indexing speed

2. We are planning to use update API to update a field, Will it reindex the whole document? Is there any option to update only a specific field without reindexing other fields?

Yes, even if you update a field, internally elasticsearch marks the old document deleted(soft deleted) and creates a new document and later on during segment merging old deleted document is actually deleted. More info on the partial update of ES is covered in detail by Elastician in this SO answer

3. It will be good if we have two index or a index with two types, one for profile other for a summary?

It depends, if while querying you need to look in both the index( types are deprecated in the latest version ), then it doesn't make sense to create two indexes, as denormalized data will give a better search performance.

Also creating two indices with a join will be a bit complex to implement, and anyway as you need to frequently update the transaction data for the author, which would have the most of the sellar data and updates on the big index is more costly, so even if you save updates on profile index but still as it would be very small, IMHO it wouldn't give much better performance but that you can test with some stress testing.