How does MySQL/InnoDB represent NULL values internally?

Question

In MySQL (or maybe I should say: with MySQL's InnoDB engine) - how are null values represented? ie how does the representation of a table (or of a single record if it's on the record level) change if a column is allowed to have NULL s?

If it's different for different column data types - either explain the variety of approaches to representing NULLs, or just pick one data type (eg INT ).

Answer 1

Reference

https://dev.mysql.com/doc/refman/5.7/en/innodb-physical-record.html

Quotes and Interpretation

ROW_FORMAT=REDUNDANT :

An SQL NULL value reserves one or two bytes in the record directory. Besides that, an SQL NULL value reserves zero bytes in the data part of the record if stored in a variable length column. In a fixed-length column, it reserves the fixed length of the column in the data part of the record. Reserving the fixed space for NULL values enables an update of the column from NULL to a non-NULL value to be done in place without causing fragmentation of the index page.

That is, 1 bit/col for NULLs, no savings in data.

ROW_FORMAT=COMPACT :

The variable-length part of the record header contains a bit vector for indicating NULL columns. If the number of columns in the index that can be NULL is N, the bit vector occupies CEILING(N/8) bytes. (For example, if there are anywhere from 9 to 15 columns that can be NULL, the bit vector uses two bytes.) Columns that are NULL do not occupy space other than the bit in this vector. The variable-length part of the header also contains the lengths of variable-length columns. Each length takes one or two bytes, depending on the maximum length of the column. If all columns in the index are NOT NULL and have a fixed length, the record header has no variable-length part.

That is, 1 bit/col, zero space for data.

I suspect, without evidence, that DYNAMIC and COMPRESSED are like COMPACT .

Column length

Each column has a 1- or 2- byte length in front of it. The choice of 1 or 2 is based on the max potential column width. (Note: Though LONGTEXT needs a 4 byte length, the 'length' is really talking about the amount that is stored in the record, not in overflow.)

Overflow storage

While I am on the topic, here is some info on what happens with "long" strings/blobs -- whether it is in the record, or stored elsewhere:

• <= 40 bytes (in a given column): Stored in the record.
• If the whole record fits in about 8KB: Store in the record.
• Otherwise, and COMPACT : 768+20 for long columns
• Otherwise, and DYNAMIC and COMPRESSED : 20 for long columns

"768" means that the first 768 bytes of the text/blob are stored in the record; "20" means a 20-byte 'pointer' to where the rest (or all) is stored.

KEY_BLOCK_SIZE controls how much column data is stored in the clustered index, and how much is placed on overflow pages.

(I am leaving REDUNDANT out of that, because I don't have the details.)

Rules of Thumb

There is 20-30 bytes of overhead for each InnoDB row.

A BTree (including the Data for InnoDB, plus each secondary index) gravitates to 69% full as blocks split, etc.

"Data_free" is woefully incomplete; don't trust it.

MyISAM is very spartan on space; it is easy to compute the space for a MyISAM table. From there, multiply by 2-3 to get the space needed for InnoDB. (There are exceptions, often involve MyISAM fragmentation, PK clustering, etc.)

Answer 2

This is for COMPACT only (REDUNDANT is interesting for historical reasons only unless you access dictionary tables). For each NULL-able column there is one bit in NULLS header.

If there are no NULL-able fields in a table NULLs header size is zero.

If a column value is NULL the bit is set and there is no value in the record data.

If a column value is not NULL the bit is unset and the column value is stored in the record's data.