Can querying a TEXT column lead to creation of a temp table?

Question

I am not very knowledgable on mysql. So forgive if its a stupid question.

I got a call from my DBA today that there is a spike in the number of temp tables my application creates.

Except that I don't create any temp table explicitly.

On troubleshooting we found that everytime a TEXT colomn is used in a sub query, a temp table was created.

As a developer I find this confusing that using a subquery and TEXT column created a temp table.

My question is, as a developer, how can I understand when a QUERY will create a temp table (in the background).

I did some googling and found that I can do something like

show GLOBAL status like 'created_tmp_disk_tables'

but what does it tell me? how can I know whether MY query caused the temp table.

Answer 1

To determine whether a statement requires a temporary table, use EXPLAIN and check the Extra column to see whether it says Using temporary.

From the documentation :

The server creates temporary tables under conditions such as these:

• Evaluation of UNION statements, with some exceptions described later.

• Evaluation of some views, such those that use the TEMPTABLE algorithm, UNION, or aggregation.

• Evaluation of derived tables (subqueries in the FROM clause).

• Tables created for subquery or semi-join materialization (see Section 9.2.1.18, “Subquery Optimization”).

• Evaluation of statements that contain an ORDER BY clause and a different GROUP BY clause, or for which the ORDER BY or GROUP BY contains columns from tables other than the first table in the join queue.

• Evaluation of DISTINCT combined with ORDER BY may require a temporary table.

• For queries that use the SQL_SMALL_RESULT option, MySQL uses an in-memory temporary table, unless the query also contains elements (described later) that require on-disk storage.

• Evaluation of multiple-table UPDATE statements.

• Evaluation of GROUP_CONCAT() or COUNT(DISTINCT) expressions.

Also,Some query conditions prevent the use of an in-memory temporary table, in which case the server uses an on-disk table instead:

• Presence of a BLOB or TEXT column in the table

• Presence of any string column in a GROUP BY or DISTINCT clause larger than 512 bytes for binary strings or 512 characters for nonbinary strings. (This applies only before MySQL 5.7.5. In addition, before MySQL 5.7.3, the limit is 512 bytes regardless of string type.)

• Presence of any string column with a maximum length larger than 512 (bytes for binary strings, characters for nonbinary strings) in the SELECT list, if UNION or UNION ALL is used

• The SHOW COLUMNS and DESCRIBE statements use BLOB as the type for some columns, thus the temporary table used for the results is an on-disk table.

Answer 2

MySQL uses temporary tables in quite a few different types of queries, without you asking for them. A few cases that often (but not always) create temporary tables:

• Subqueries
• GROUP BY
• UNION
• Querying through a VIEW
• Common table expressions ( WITH ... syntax)

A more complete guide to when MySQL uses "internal" temporary tables is found in the documentation: https://dev.mysql.com/doc/refman/5.7/en/internal-temporary-tables.html

Basically, a temp table is used when the execution of query needs to save rows from a partial result, and then move on to refine that partial result into a final result.

Temp tables are fairly expensive for the MYSQL Server to use, so we like to avoid them when possible. Especially costly is when the temp table is written to the filesystem instead of remaining in memory. This happens when the temp table involves TEXT or BLOB columns, or if it needs to store more rows than can fit in an amount of memory limited by the tmp_table_size configuration option.

You can (mostly) tell when your query will use a temp table by analyzing the query with EXPLAIN . See https://dev.mysql.com/doc/refman/5.7/en/explain.html

Sometimes internal temp tables are unavoidable, because there's no other way to get the query result you need. But sometimes you can rewrite your SQL query or use indexes to help the query get the same result without using a temp table. It depends on the specific query, and your table definitions. Query optimization is a process that must be done on a case-by-case basis.