从API处理大数据以可视化

Question

Let's say I have an API that gives me the values of stock for the last month. The data is sampled every hour.

Now I want to make a web app that would visualize this data on a line chart. I don't need all the hourly samples, so my question is how should I make this work?

My idea is that there would be a backend app (ie in Java Spring) that would GET the data from the API and calculate the average for each day (using a stream, maybe parallel stream?) and then put that in a new collection and pass it on to the front end to put in a chart.

Answer 1

Start thinking from the UI, what do you need there, how often do you need it and how fast ?

Then get the data from the backend, if there is too much data at once and the API cannot do otherwise, either:

• get data and reduce to what the UI needs (backend), use once and throw away
• OR get data and reduce to what the UI needs (backend), keep in cache for a while
• OR pre-process the data so that when the UI needs it, it will be ready

For the return format, consider something lightweight, like some simple named json array {"dayAverages": [0.34, 1253.432, ...]}, "month" : 2, "year": 2018} , then in the UI adapt to the needs of your lib (that is debatable).

Also observe how users use the UI, then you may get some ideas on how to optimize the experience (preload next month ...)

If you do this for learning purposes, consider doing it async + lambdas = bonus :)

Answer 2

As to your question "...how should I make this work?" --

This is extremely broad. There are many, many ways to do this. Some of these ways depend heavily on your architecture, how much traffic is expected to your app, what request-load the API can handle, etc. Here are a few general things to consider:

• Any sort of MVC architecture (or similar) would be a good fit for your Web app.
• You mention needing a "backend app" of some type. Not sure what you mean here, but the averaging features can be built directly into your Web app framework without needing a separate back-end app.
• If you're going to calculate averages for display in the Web app, you will need to maintain state somewhere. Assuming the API doesn't give this to you, you'll need a database of some type, or at least some type of memory caching storage engine to facilitate this. How you do this will depend on your architecture and the traffic/load on your app (eg will you have multiple, load-balanced servers).

Hope that helps. We could give more if you ask some specific questions.