I want to search Disallow: /
in robots.txt of domains.
I have wrote regex, but it is not working.
if(preg_match("!Disallow:\s*/\s\r\n!i",$string,$disallow_char))
{
print_r($disallow_char);
}
Following are two test cases.
1)
User-agent: *
Disallow: /
2)
User-agent: *
Disallow: /product/generate_pdf/40
Disallow: /news/
Disallow: /news/bollards
Disallow: /product/generate_pdf/44
Disallow: /
Disallow: /page_management/insert
Disallow: /glossary/ajax_call/update_words
It should output true for both the cases.
You need to assert that either a newline sequence or the end of the string follows:
echo preg_match('~Disallow:\h*/(?:\R|$)~i', $string)
Explanation :
Disallow: # 'Disallow:'
\h* # horizontal whitespace (0 or more times)
/ # '/'
(?: # group, but do not capture:
\R # '\R' (any Unicode newline sequence)
| # OR
$ # before an optional \n, and the end of the string
) # end of grouping
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.