使用语言代码禁止robots.txt中的子目录

Question

We have a website translated to about 15 languages, with more comming in regulary from our editors.

Our URL is built up as follows:

SiteName.Company.com/{LCID}/Folder1/Folder2/

Our company want's to hide a subfolder/page from the search engines.

LCID can be anything from EN-US to ZH-HANS. And i am looking for a solution where we don't have to update the robots file every time a new language is added.

Can i make a wildcard entry or filter out a single foldername regardless of it's path?

IE: companyinfo.nokia.com/en-us/aboutus/darknews/ I want Google to crawl everything but darknews and down (so /darknews/ELOPSellsOut or /darknews/3210revival)

Is this possible?

Answer 1

You can do this with wildcards:

User-agent: *
Disallow: /*/darknews/

Just be aware that not all robots support wildcards. This will work for Googlebot and Bingbot, but it will not work for many others.