What are maximum limits of Mariadb columnstore?

Question

I want to create wide table of thousands of rows in mariadb columnstore. I didn't find any documentation of max number of columns allowed by storage engine. I would also like to know how will be the performance of columnstore for 1000 integer columns?

Answer 1

(Caveat: This 'Answer' is based on my understanding of the design, not on any 'facts'.)

The disk footprint of a table should be proportional to the number of columns.

As with most things in MariaDB, there is probably a hard limit on the number of columns, but I can think of no reason for it to be under 1000. Perhaps, instead, some larger power of 2.

When referencing only a small number of columns, it should not matter how many columns there are in the table. The way the data is structured should allow fetching each column with a relatively fixed amount of effort.

For filtering, I would expect the effort taken to depend on the number of columns used for filtering, and their distribution. If your WHERE clause references a lot of columns, I would not expect good performance.

With any Engine, having lots of columns is not necessarily a wise design. In general, when you have lots of columns that are not used for filtering or sorting ( WHERE , ORDER BY ), you may as well toss them into a JSON string (or other structure) and store as a single TEXT or BLOB columns. Then let the application parse the string to get the individual columns.

Columnstore shines for 'filtering'. It also is very good in compressing data, and my JSON suggestion would defeat this. But now you are into speed-vs-space tradeoffs that are very data-specific.

Would you care to describe your proposed dataset?

Answer 2

There is no limitation on number of columns for Columnstore but data ingestion performance is not the best ATM. We are decreasing the timing significantly in the nearest future.

Answer 3

When I tried to create a ColumnStore table which has 2310 columns, it returned "Error Code: 1117. Table definition is too large".

I decreased the number of columns and tried again. It looks like that the maximum number of columns for MariaDB ColumnStore is 2201.