EDIT: Doing this with robots.txt isn't good enough for me.
This code block entire site from google. I do not know how to keep this code functional but allow google to access one folder on the site.
RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} Googlebot [OR]
RewriteRule ^(.*)$ - [F,L]
Does this works if i had 2 conditions?
Options +FollowSymLinks -MultiViews
# Turn mod_rewrite on
RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} Googlebot [OR]
RewriteCond %{HTTP_USER_AGENT} yahoo [OR]
RewriteCond %{REQUEST_URI} !^/allowed-folder/ [NC]
RewriteRule ^ - [F,L]
As suggested in comments create a file robots.txt
under DOCUMENT_ROOT with following content:
User-Agent: googlebot
Disallow: /
Allow: /allowed-folder/
If you really want to handle this via mod_rewrite then enable mod_rewrite and .htaccess through httpd.conf
and then put this code in your .htaccess
under DOCUMENT_ROOT
directory:
Options +FollowSymLinks -MultiViews
# Turn mod_rewrite on
RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} Googlebot [NC]
RewriteCond %{REQUEST_URI} !^/allowed-folder/ [NC]
RewriteRule ^ - [F,L]
create file with name robots.txt
insert
User-agent: *
Disallow: /folder you dont wont google inside in/
Disallow: /other folder you dont wont google inside in/
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.