MySQL Temp Table Location

Question

I'm getting this error when running a query:

[ERROR] [MY-013132] [Server] The table '/tmp/#sql1127b_9_0' is full!

I go into the /tmp directory but I can't see the table, I'm presuming it gets deleted when the query has finished running.

So I go into the directory when the query is running (the query takes a few minutes to run - against 40 million records). I don't see the table. I refresh the directory, I still don't see it. How can I see it? I want to see it and establish its location so I can figure out why it's filling up when I have lots of free diskspace.

I'm using MySQL 8.0.23 - I previously used MySQL 5.7.33 on the same machine with the same database and same query, never got a problem.

This is my diskspace:

Filesystem      Size  Used Avail Use% Mounted on
tmpfs           3.2G  3.7M  3.2G   1% /run
/dev/sdc3       215G   38G  167G  19% /
tmpfs            16G     0   16G   0% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           4.0M     0  4.0M   0% /sys/fs/cgroup
/dev/sdb6       107G   81G   21G  80% /media/Kingston_SSD_120GB
/dev/sda        459G  335G  101G  77% /media/Hitachi
/dev/sdc2        33M  7.8M   25M  24% /boot/efi
tmpfs           3.2G  120K  3.2G   1% /run/user/1001

I didn't set any diskspace for /tmp - as can be seen, I have 167GB free space on / .

Answer 1

MySQL uses a trick that has been part of POSIX systems forever. It opens the temp file, and immediately unlinks it. Therefore it's not viewable in any directory listing. But POSIX systems like UNIX and Linux shouldn't actually remove an unlinked file while a process has an open file handle to it. So once the query using the temp table finishes, it will close the file handle, and then the OS will automatically remove the file and free the storage it was using.

This is generally better than requiring the server code remember to remove the tempfile when it's done with it. It also accounts for like the thread terminating, or mysqld crashing. At least it won't leave stale temp files littering your filesystem.

You can view the size of unlinked files with lsof -s . I'll leave it to you to look up examples of how to use that command (Google is your friend here).

It's thinly possible that a temp file uses up your 167GB of free space.

Or it could be that the temp file only uses 8GB, but you may have 20 threads doing the same query at the same time. I have seen that happen once.

But it's probably more likely that you have a value of tmp_table_size that is constraining the size of the temp table.

If you hit the limit, you can raise that configuration option, either as a session variable when you need it, or globally in the my.cnf .

But I would first try to optimize the query. Why does it need to create such large temp tables? Could it be optimized to examine fewer rows, or perhaps avoid creating a temp table altogether?

Answer 2

from the official documentation:

On Unix, MySQL uses the value of the TMPDIR environment variable as the path name of the directory in which to store temporary files. If TMPDIR is not set, MySQL uses the system default, which is usually /tmp, /var/tmp, or /usr/tmp.

so if you didnt configure TMPDIR, then your file is needed to be under /tmp, /var/tmp, or /usr/tmp

IF you are getting this error instantly (within a second or two), I suspect that this is a permission issue. But otherwise, It looks like the temporary table really filling the disk.

EDIT: try looking for the file while executing the query. Because The file is getting deleted after the SQL error

Also this post can help you

How can I limit the size of temporary tables?