网址重写和robots.txt

Question

I have made urls SEO friendly ie

http://mydomain.com/topic/title-of-page

Previously, the url of the above page was

http://mydomain.com/search?id=6567889

Now Google is still showing the second URL in the search results. My question is that if i disallow /search in robots.txt , will Google stop crawling that page altogether or it will still crawl mydomain.com/topic/title-of-page ie the new URL?

Thanks a lot for your help. Sorry for the spaces in the URL as SO wouldn't let me post them

Seb

Answer 1

You'll want to fix this right away. There are going to be issues with duplicated content on the site in regard to SEO.

To cut half of the search engine traffic from your site, add this to your robots.txt file:

User-agent: *
Disallow: /search.php

That way none of the search engines will mine the old link. They may still have it cached for their search results. That being said you'll need to handle each of the pages they serve with a 301 redirect so you're not penalized. This will tell them to update their cache.

Additionally you'll want to add a sitemap.xml file that contains all for the proper URLs on the server in order of priority.

Assuming you have the new path stored in the DB, you'll need to do something like this to search.php . That way you can keep your URLs for SEO. 301 is a Permanent Redirect.

if(preg_match('!^/search!',$_SERVER['REQUEST_URI'])){
  $tempID = $_GET['id'];
  $tempID = mysqli_real_escape_string($tempID);

  $query = "select count(*) from table where id=$tempID";

  $count = mysqli_result(mysqli_query($db_link,$query),0);

if($count==1){

 $query2 = "select correctPath  from table where id=$tempID";

 $correctPath = mysqli_result(mysqli_query($db_link,$query2),0);
   header( "HTTP/1.1 301 Moved Permanently" );
   header( "Location: $correctPath" );
   exit();
  } else {
  //non-existent ID
   header( "HTTP/1.1 301 Moved Permanently" );
   header( "Location: /" );
   exit();
  } 
}

Otherwise you could replace the second query with the function you're using to build the paths.

You could add something to the .htaccess file like this for every entry, but it's not future proof and the larger the list, the longer to process (more server overhead).

RewriteEngine On
RewriteRule ^search\.php\?id=1  /page1_new_path/ [L,R=301]
RewriteRule ^search\.php\?id=2  /page2_new_path/ [L,R=301]