Logstash + ElasticSearch + Kibana combine results from different fields in different documents

Question

We have Apache log analyzed by Elasticsearch (2.1.0) and Kibana (4.3.0). Logs are parsed and shipped to Elasticsearch by Logstash running on web servers and reading Apache combined log format. All works good but now we need analyze more complicated pattern. We have documents with field “purchase_id” which has integer value (like 130012, 130016, 133552 etc). We have OTHER documents which have integer field “view_id” with same values (like 130012, 130016, 133552 etc.)

Both fields never appear in same document, because those fields extracted from different URI in Apache log. Our goal is calculate and visualize percentage of appearance in given time frame of values in “purchase_id” compared to values in “view_id”. For example, lets say we want to see current purchase rate of item 130012. It may appear in last 30 seconds 1000 times in documents with field “purchase_id” and in same last 30 seconds it may appear 40000 times in documents with field “view_id”. This is obvious because only small amount of people buy item compared to amount of people exposed to product. I need to calculate and visualize that in time frame there was 1000 times purchase_id of item 130012 and 40000 times view_id of item 130012 then divide 1000 by 40000 and multiply 100% so I get 2.5% visualized on dashboard (for item 130012). Of course I have many such purchase_id=view_id=(some number):int pairs, so I need calculate percentage for all of them and display, lets say 20 with highest percentage. This will allow me know the best selling items compared to advertisements we invest.

Answer 1

我会为kibana跟踪此问题 。