Send sensor data to hazelcast database and link it to development tools to design dashboards

Question

Can we use hazel-cast database to link and design the data according to tracker with bar graph, below are the points which I need to confirm to build the application for hardware: - I am using temperature sensor interfacing with Arduino Yun and wanted to upload the data given by temperature sensor on hazel-cast server.

• By using single database output uploaded in hazelcast server, reads the data through database through Arduino MKR1000.

• Link the data to different development tools to design different types of dashboards like Pie chart, Bar chart, Line chart etc. Please suggest how the best way to link to create the database in data-grid

Answer 1

You can configure Hazelcast to use a MapLoader - MapStore to persist the cached data to any back-end persistence mechanism – relational or no-sql databases may be good choices. On your first point, I wouldn't expect anything running on the Arduino to update the database directly, but the MKR1000 is going to get you connectivity, so you can use Kafka/MQTT/… - take a look at https://blog.hazelcast.com/hazelcast-backbone-iot-internet-things/ . If you choose this route, you'd set up a database that is accessible to all cluster members, create the MapLoader/MapStore class (see the example code , for help) and configure the cluster to read/write. Once the data is in the cluster, access is easy and you can use a dashboard tool of your choice to present the data.

(edit) - to your question about presenting historical data on your dashboard:

Rahul's blog post describes a very cool implementation of near/real-time data management in a Hazelcast RingBuffer . In that post, I think he mentioned collecting data every second and buffering two minutes worth.

The ring buffer has a configured capacity, but note that he is over-writing, on add - this is kind of a given for real-time systems; given the choice is losing older data or crashing.

For a generalized query-tool approach, I think you'd augment this. Off the top of my head, I could see using the ring-buffer in conjunction with a distributed map. You could (but wouldn't need to) populate the map, using an map-event interceptor to populate the ring buffer. That should leave the existing functionality intact. The map, though, would allow you to configure a map-store/map-loader, so that your data is saved in a backing store. The map would support queries - but keep in mind that IMDG queries do not read through to the backing store.

This would give you flexibility, at the cost of some complexity. The real-time data in the ring buffer would be always available, quickly and easily. Data returned from querying the map would be very quick, too. For 'historical' data, you can query your backing-store - which is slower, but will probably have relatively great storage capacity. The trick here is to know when to query each. The most recent data is a given, with it's fixed capacity. You need to know how much is in the cluster - ie how far back your in-memory history goes. I think it best to configure the expiry to a useful limit and provision the storage so that data leaves the map by expiration - not eviction. In this way, you can know what the beginning of the in-memory history is. Monitoring eviction events would tell you that your cluster has a complete view of data back to a known time.

Answer 2

How you want to use data on your dashboard will basically depend on how you have modelled your data - one map or multiple maps etc. Then you can retrieve data through single key-based lookups or by running queries and use that for your dashboard. You can define the lifetime of the data - be it few minutes or hours or days. See eviction: http://docs.hazelcast.org/docs/3.10.1/manual/html-single/index.html#map-eviction

If you decide to use a visualisation tool for dashboard that can use JMX then you can latch on to Hazelcast exposed JMX beans that would give you information about data stored in the cluster and lot more. Check out this: http://docs.hazelcast.org/docs/3.10.1/manual/html-single/index.html#monitoring-with-jmx