How to add Significance Testing and Correlation calculations into the cube.js flow?

Question

We are currently using cube.js for its data infrastructure, querying and API capabilities.

However as part of our requirements, we need to perform statistical calculations like significance testing and correlations on certain measures. Is there an in-built method of doing this in cube.js?

We have tried using the correlation function built into PostgreSQL by calling it from the sql field in the cube.js schema and it works (code shown below) however are there any alternatives?

measures: {
        testCorrelation: {
          title: 'Test Correlation',
          sql: `corr(${CUBE}.col1,${CUBE}.col2)`,
          type:`number`
        }
}

For significance testing, we haven't found any solutions yet.

We considered creating a python statistical microservice which would make requests to the cubejs server (aka be a cube client), take care of the calculations and then send the resultSet with the calculations appended to the react client. Would this flow/architecture work and has similar work been done in this area?

Would appreciate any responses/ideas. Thank you.

Answer 1

Is there an in-built method of doing this in cube.js?

Not and probably will not.

Cube.js provides an ability to define SQL inside measure. Some tasks can be solved on top of built-in functions. Another part can be done on top of user-defined procedures/functions or installed extensions.

For significance testing, we haven't found any solutions yet.

PostgreSQL its self has only limited stats functions built-in, but the PL/R extension supports more statistical tests and other features than you could ever want.

We considered creating a python statistical microservice which would make requests to the cubejs server (aka be a cube client)

It's better to do aggregations in the database, because:

• you will not able to use pre-aggregations
• data transfer can take a lot of time & memory
• some databases can use columnar format for storing and SIMD execution.

Thanks