使用Java从mssql表中检索直方图

Question

I want to implement java application that can connect to any sql server and load any table from it. For each table I want to create histogram based on some arbitrary columns.

For example if I have this table

name   profit
------------
name1   12
name2   14
name3   18
name4   13

I can create histogram with bin size 4 based on min and max value of profit column and count number of records for each bin.

result is:

profit    count
---------------
12-16     3
16-20     1

My solution for this problem is retrieving all the data based on required columns and after that construct the bins and group by the records using java stream Collectors.groupingBy .

I'm not sure if my solution is optimized and for this I want some help to find the better algorithm specially when I have big number of records.(for example use some benefits of sql server or other frameworks that can be used.)

Can I use better algorithm for this issue?

edit 1: assume my sql result is in List data

private String mySimpleHash(Object[] row, int index) {
        StringBuilder hash = new StringBuilder();
        for (int i = 0; i < row.length; i++)
            if (i != index)
                hash.append(row[i]).append(":");
        return hash.toString();
    }
 //index is index of column for histogram
List<Object[]> histogramData = new ArrayList<>();
final Map<String, List<Object[]>> map = data.stream().collect(
                Collectors.groupingBy(row -> mySimpleHash(Arrays.copyOfRange(row, index))));
for (final Map.Entry<String, List<Object[]>> entry : map.entrySet()) {
   Object[] newRow = newData.get(rowNum); 
   double result = entry.getValue().stream()
                                .mapToDouble(row -> 
   Double.valueOf(row[index].toString())).count();
   newRow[index] = result;
   histogramData.add(newRow);
}

Answer 1

As you have considered, performing the aggregation after getting all the data out of SQL server is going to be very expensive if the number of rows in your tables increase. You can simply do the aggregation within SQL.

Depending on how you are expressing your histogram bins, this is either trivial or requires some work. In your case, the requirement that the lowest bin start at min value requires a little bit of setup as opposed to binning starting from 0. See sample below. The inner query is mapping values to a bin number, the outer query is aggregating and computing the bin boundaries.

CREATE TABLE Test (
    Name varchar(max) NOT NULL,
    Profit int NOT NULL
)

INSERT Test(Name, Profit)
VALUES
('name1', 12),
('name2', 14),
('name3', 18),
('name4', 13)

DECLARE @minValue int = (SELECT MIN(Profit) FROM Test)
DECLARE @binSize int = 4

SELECT
    (@minValue + @binSize * Bin) AS BinLow,
    (@minValue + @binSize * Bin) + @binSize - 1 AS BinHigh,
    COUNT(*) AS Count
FROM (
    SELECT
    ((Profit - @minValue) / @binSize) AS Bin
    FROM
    Test
) AS t
GROUP BY Bin


| BinLow | BinHigh | Count |
|--------|---------|-------|
|     12 |      15 |     3 |
|     16 |      19 |     1 |

http://sqlfiddle.com/#!18/d093c/9