使用 PHP 在 robots.txt 中搜索连续的“用户代理”指令

Question

With PHP, I want to check (true/false) if there are consecutive 'User-agent' directive in robots.txt file.

With this regexp, preg_match('~User-agent:\\h*(?:\\R|$)~i', $string) I found all 'User-agent:' line but I haven't found how to detect consecutive lines.

User-agent:    # 'User-agent:'
\h*            # horizontal whitespace (0 or more times)
(?:            # group, but do not capture:
  \R           #   '\R' (any Unicode newline sequence) 
 |             #  OR
  $            #   before an optional \n, and the end of the string
)              # end of grouping

For example

User-agent: 008
user-agent: Accoona
User-Agent: Googlebot
User-Agent: aipbot*
disallow: /

Result: True

User-Agent: Googlebot
Crawl-delay: 60
User-agent: aipbot*
disallow: /

Result: False

User-agent: 008
Crawl-delay: 2
user-agent: Accoona
User-Agent: Googlebot
User-Agent: aipbot*
disallow: /

Result: True

Answer 1

This may seem a derpy answer, but why not just repeat your regex? Surely User-agent:\\h*(?:[a-zA-Z0-9\\*]*\\R|$)User-agent:\\h*(?:[a-zA-Z0-9\\*]*\\R|$) only matches if there are two consecutive user agents?

https://regex101.com/r/ximRMo/1

Add/remove non-user-agent lines between the consecutive one, 0 matches. Two consecutive lines cause a match.

Answer 2

Without regex:

$filePath = 'robots.txt';

try {
    if ( false === $fh = fopen($filePath, 'rb') )
        throw new Exception('Could not open the file!');

} catch (Exception $e) {
    echo 'Error (File: ' . $e->getFile() . ', line ' . $e->getLine() . '): ' . $e->getMessage();
}

var_dump(hasSuccessiveUA($fh));

fclose($fh);    

function hasSuccessiveUA($fh) {
    $previous = false;

    while ( false !== $line = fgets($fh) ) {
        $current = ( stripos($line, 'user-agent:') === 0 );
        if ( $previous && $current ) return true;
        $previous = $current;
    }

    return false;
}

Advantage: when the answer is true, you don't have to load the file until the end.