What data do the MariaDB system variables refer to e.g. innodb_log_buffer_size

Question

Many of the MariaDB system variables define maximum data sizes and I'm not clear about what data exactly is being measured, in the context of write operations where I'm sending a lot of data over JDBC, eg:

innodb_buffer_pool_size = 24696061952

and

innodb_log_buffer_size = 8388608

and

max_allowed_packet = 562036736

I use a benchmark where I insert, update or delete 50000 rows with 2MB of raw data, that is date and double data.

So when MariaDB implements these variable-defined limits, does it look at the raw value of the parameters send down the wire, or does it include all the SQL string as well?

ie is it measuring simply

2012-01-01,0.1234
2012-01-02,0.4321
2012-01-03,0.9999
...

or does it consider the bytes in the SQL string:

UPDATE data SET value = 0.1234 WHERE date = '2012-01-01'
UPDATE data SET value = 0.4321 WHERE date = '2012-01-02'
UPDATE data SET value = 0.9999 WHERE date = '2012-01-03'

which in this example is obviously 2 to 3 times larger?

Answer 1

Innodb_log_buffer_size

It refers to something internal. transactions are logged into redo log (ib_logfile0/ib_logfile1). redo log records are buffered in the redo log buffer, until the transaction commits, The format of the redo log records is not documented, and also subject to change between the versions. You can see how much is written into the log by watching changes of the innodb_os_log_written status variable.

 MariaDB [test]> show global status like 'innodb_os_log_written';
 +-----------------------+-------+
 | Variable_name         | Value |
 +-----------------------+-------+
 | Innodb_os_log_written | 12800 |
 +-----------------------+-------+
 1 row in set (0.00 sec)

innodb_buffer_pool_size

You data is tiny, and buffer pool is big enough. Buffer pool is how innodb caches the data. If you data is bigger than bufferpool , which is not in your case, Innodb would need to read from the disk more often. So leave it as is, or you can make it smaller.

max_allowed_packet

It is just a security measure, trying to prevent DDOS. If one sends very large packets, they make server allocate more memory, and perhaps run out of memory. But if you do not send large packets, server does not allocate a lot of memory, and if you're not afraid of DDOS, you can have it as big as 1G.

To get the idea how much data was sent or received on the server side, you can watch the status variable

MariaDB [test]> show status like 'bytes%';
+----------------+-------+
| Variable_name  | Value |
+----------------+-------+
| Bytes_received | 570   |
| Bytes_sent     | 1716  |
+----------------+-------+
2 rows in set (0.00 sec)

How does this correspond to the size of data you send? There is not always a straightforward answer.

• Statements that are not (server-side) prepared

  If you just send unprepared queries, you'll send 4 bytes packet header + 1 byte for command (COM_QUERY) + your SQL query string. if you query is larger than 16M, it will be split into multiple packets, and there is a 4 byte overhead per packet.

• Statements that are server-side prepared.

  With server-side prepared statements it is more complicated. Every datatype has its own encoding. , so ints are not sent as text, nor floats, nor dates. strings are still sent as strings. Usually, the amount of data sent will be a little bit smaller (you do not send SQL commands after all, just data).

• Batch specific optimizations

  MariaDB 10.2 server and JDBC also have an optimization to send multiple data for a single prepared statement. This can reduce the amount of data sent by client (a little), but amount of data received by the client will be significantly reduced, it is just single packet for all commands, not one-per-command. This optimization does not work for DELETE, yet.

For batching of statements that are not server-side prepared, JDBC has yet another optimization, which transforms a lot of similar INSERT queries into one multi-insert, which also reduces the amount of sent and received bytes, but this is INSERT only (there is now a task to do something similar for DELETEs)