PostgreSQL could not open file "base/xxxx/xxxxx" no such file or directory

Question

Recently I had a Hardware failure on my linux machine and after fixing hardware problem and bringing back my linux machine up, when I execute query against one of my table, following error is returned.

ERROR: could not open file "base/17085/281016": No such file or directory.

When checked in postgresql/base/17085 directory, file 281016 does not exist.

Would the issue be resolved if I create file manually by using below commands? or is it a bad approach causing more troubles in future?

#touch 281016
#chown postgres:postgres 281016
#chmod 600 281016

Answer 1

Short answer: restore from backup. Then investigate your setup, you are running an unsafe system.

Long answer:

Assuming you don't have a backup but need the database you have learnt a valuable lesson. Take and check your backups.

If it's a simple SELECT * FROM bad_table that's failing then it's the table that has the problem. If not, COPY the data out immediately, you've been lucky and it's just an index that got destroyed.

Then dump all the rest of your tables.

Then do some checks to make sure the data is in a sane state before restoring it and putting it back in production.

Now - barring bugs in PostgreSQL (unlikely) this should be impossible. Since we are talking about a missing file I would guess your disks are reporting data is flushed and synced when it really isn't. Check your fsync settings in postgresql.conf anyway.

Answer 2

In my local server, it was an issue with one of my test databases, just drop this database. It is ok now. For prodution server isn't a good idea droping database.

Answer 3

I had the same problem running vacuum on Postgres 12.

This database is production, but fortunately not customer-facing. It does track storage details for 3 large Isilon clusters that are important and I needed to recover it. This is what fixed my system.

After seeing errors:

database is not accepting commands to avoid wraparound data loss in database ((database_2_name)) 

HINT: Stop the postmaster and vacuum that database in single-user mode. You might also need to commit or roll back prepared transactions, or drop stale replication slots.

I stopped the postmaster (CentOS7 + Systemd)

systemctl stop postgresql-12.service

Then started postgres in single user mode:

# su - postgres
postgres --single -D /var/cache/insightiq/pgsql/data  ((database_1_name))

The database kicks out a little warning:

> WARNING:  database with OID 16403 must be vacuumed within 990276 transactions
> HINT:  To avoid a database shutdown, execute a database-wide VACUUM in that database.
         You might also need to commit or roll back old prepared transactions, or drop stale replication slots.

PostgreSQL stand-alone backend 12.4
backend> 

Then attempting the vacuum, but ran into the missing number? under my database's OID (16403). I noticed this 16403 was associated at the top level of the storage set up for my ((database_1_name))

backend> vacuum

<2023-01-05 19:00:41.150 UTC [4229] > ERROR:  could not open file "base/16403/2613": No such file or directory
<2023-01-05 19:00:41.150 UTC [4229] > STATEMENT:  vacuum

As the original poster suggested I did attempt to add a file at the location (where the data should be), but there was no change in the error when I tried vacuum again. The same 'could not open file "base/16403/2613"'

In my case, my large database structures are off on NFS nad I could see them changing when I ran vacuum. I searched my local filesystem for anything named 'base/16403' and found a directory structure where the data was not expected to be, in /var/cache/insightiq/pgsql/data/base/16403.

With no other choice I decided to GAMBLE! I created an empty file in the location vacuum was complaining about (cache location instead of data location)

# ls /var/cache/insightiq/pgsql/data/base/16403/2613
ls: cannot access /var/cache/insightiq/pgsql/data/base/16403/2613: No such file or directory
# touch /var/cache/insightiq/pgsql/data/base/16403/2613
# chown postgres:postgres /var/cache/insightiq/pgsql/data/base/16403/2613

Presto. Single-user vacuum works on my database. The vacuum recommended doing the same on the other databases, which went smoothly. At the end, I stop/started everything and this System is working again.