Want empty folder to return 404 instead of 403

Question

I use the following code in my httpd.conf file to block unwanted bots and visitors (eg hitting non existing wp-login pages):

SetEnvIf User-Agent BadBot GoAway=1
Order allow,deny
Allow from all
Deny from env=GoAway

This will give them a 403 Forbidden error. On my custom 403 page I save the IP's to a database for permanent blocking / disabling them from using contact forms etc.

This is working very well. However I noticed that some users are being blocked because they visited an empty folder. This is unwanted.

Using Options -Indexes I have prevented directory viewing but this will also output a 403 Forbidden error.

Question: How to serve and display a 404 error when visiting an empty folder instead of a 403?

Answer 1

Since you appear to have access to httpd.conf then you can do something like the following in a server config or virtualhost context:

RewriteEngine On
RewriteCond %{LA-U:REQUEST_FILENAME} -d
RewriteRule /. - [R=404]

Alternatively, near the top of your .htaccess file (since you've tagged your question .htaccess ), include the following (this will also work in a directory context):

RewriteEngine On
RewriteCond %{REQUEST_FILENAME} -d
RewriteRule . - [R=404]

The purpose of the pattern /. (or . ) is to prevent a 404 being served for the document root.

Depending on where you define your custom ErrorDocument , you may get a different 404 response for each method.

---

Alternatively, in your custom 403, perform the same check for the "BadBot" and only log the entry if there is a match. Or, only log the entry when the request does not map to a directory.