解析亞馬遜 s3 日志文件 (PHP)

Question

我希望解析以空格分隔的亞馬遜 s3 日志文件。 唯一的問題是，一些以空格分隔的字段包含空格。 我將如何解析這樣的文件？

450227804f8fd31c931036b020ddd0003a03b421d8c669d8858c7c15d72c renderd [10/Apr/2014:19:32:23 +0000] 75.256.56.200 450227804f8fd31c931036b020000343afa03b421d8c669d8858c7c15d72c 0231400AA3D3533C REST.GET.OBJECT Trailer.mp4 "GET /Trailer.mp4?AWSAccessKeyId=AKIAJFV33YRQMN63AQCQ&Expires=1397159234&Signature=8ipN9ymsB5gCzxChTu9lD6ZMrdA%3D HTTP/1.1" 206 - 5016183 16149754 216682 39 "http://example.com" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36" -

Answer 1

您可能可以使用正則表達式來解析日志文件以獲取各個部分

這是PHP中的一個例子來做到這一點

<?php 
$string ='450227804f8fd31c931036b020ddd0003a03b421d8c669d8858c7c15d72c renderd [10/Apr/2014:19:32:23 +0000] 75.256.56.200 450227804f8fd31c931036b020000343afa03b421d8c669d8858c7c15d72c 0231400AA3D3533C REST.GET.OBJECT Trailer.mp4 "GET /Trailer.mp4?AWSAccessKeyId=AKIAJFV33YRQMN63AQCQ&Expires=1397159234&Signature=8ipN9ymsB5gCzxChTu9lD6ZMrdA%3D HTTP/1.1" 206 - 5016183 16149754 216682 39 "http://example.com" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36" -';

$pattern = '/(?P<owner>\S+) (?P<bucket>\S+) (?P<time>\[[^]]*\]) (?P<ip>\S+) (?P<requester>\S+) (?P<reqid>\S+) (?P<operation>\S+) (?P<key>\S+) (?P<request>"[^"]*") (?P<status>\S+) (?P<error>\S+) (?P<bytes>\S+) (?P<size>\S+) (?P<totaltime>\S+) (?P<turnaround>\S+) (?P<referrer>"[^"]*") (?P<useragent>"[^"]*") (?P<version>\S)/';

preg_match($pattern, $string, $matches);
print_r($matches);

Answer 2

我稍微修改了 Jeremy Quinton 的答案，以便更好地匹配

  <?php 
  $string ='450227804f8fd31c931036b020ddd0003a03b421d8c669d8858c7c15d72c renderd [10/Apr/2014:19:32:23 +0000] 75.256.56.200 450227804f8fd31c931036b020000343afa03b421d8c669d8858c7c15d72c 0231400AA3D3533C REST.GET.OBJECT Trailer.mp4 "GET /Trailer.mp4?AWSAccessKeyId=AKIAJFV33YRQMN63AQCQ&Expires=1397159234&Signature=8ipN9ymsB5gCzxChTu9lD6ZMrdA%3D HTTP/1.1" 206 - 5016183 16149754 216682 39 "http://example.com" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36" -';

  $pattern = '/(?P<owner>\S+) (?P<bucket>\S+) \[(?P<time>[^]]*)\] (?P<ip>\S+) (?P<requester>\S+) (?P<reqid>\S+) (?P<operation>\S+) (?P<key>\S+) "(?P<method>[^ ]*) (?P<path>[^"]*)" (?P<status>\S+) (?P<error>\S+) (?P<bytes>\S+) (?P<size>\S+) (?P<totaltime>\S+) (?P<turnaround>\S+) "(?P<referrer>[^"]*)" "(?P<useragent>[^"]*)" (?P<version>\S)/';

  preg_match($pattern, $string, $matches);
  print_r($matches);

  ?>

  result : 
  Array
  (
      [0] => 450227804f8fd31c931036b020ddd0003a03b421d8c669d8858c7c15d72c renderd [10/Apr/2014:19:32:23 +0000] 75.256.56.200 450227804f8fd31c931036b020000343afa03b421d8c669d8858c7c15d72c 0231400AA3D3533C REST.GET.OBJECT Trailer.mp4 "GET /Trailer.mp4?AWSAccessKeyId=AKIAJFV33YRQMN63AQCQ&Expires=1397159234&Signature=8ipN9ymsB5gCzxChTu9lD6ZMrdA%3D HTTP/1.1" 206 - 5016183 16149754 216682 39 "http://example.com" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36" -
      [owner] => 450227804f8fd31c931036b020ddd0003a03b421d8c669d8858c7c15d72c
      [1] => 450227804f8fd31c931036b020ddd0003a03b421d8c669d8858c7c15d72c
      [bucket] => renderd
      [2] => renderd
      [time] => 10/Apr/2014:19:32:23 +0000
      [3] => 10/Apr/2014:19:32:23 +0000
      [ip] => 75.256.56.200
      [4] => 75.256.56.200
      [requester] => 450227804f8fd31c931036b020000343afa03b421d8c669d8858c7c15d72c
      [5] => 450227804f8fd31c931036b020000343afa03b421d8c669d8858c7c15d72c
      [reqid] => 0231400AA3D3533C
      [6] => 0231400AA3D3533C
      [operation] => REST.GET.OBJECT
      [7] => REST.GET.OBJECT
      [key] => Trailer.mp4
      [8] => Trailer.mp4
      [method] => GET
      [9] => GET
      [path] => /Trailer.mp4?AWSAccessKeyId=AKIAJFV33YRQMN63AQCQ&Expires=1397159234&Signature=8ipN9ymsB5gCzxChTu9lD6ZMrdA%3D HTTP/1.1
      [10] => /Trailer.mp4?AWSAccessKeyId=AKIAJFV33YRQMN63AQCQ&Expires=1397159234&Signature=8ipN9ymsB5gCzxChTu9lD6ZMrdA%3D HTTP/1.1
      [status] => 206
      [11] => 206
      [error] => -
      [12] => -
      [bytes] => 5016183
      [13] => 5016183
      [size] => 16149754
      [14] => 16149754
      [totaltime] => 216682
      [15] => 216682
      [turnaround] => 39
      [16] => 39
      [referrer] => http://example.com
      [17] => http://example.com
      [useragent] => Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36
      [18] => Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36
      [version] => -
      [19] => -
  )

Answer 3

亞馬遜現在已將更多字段附加到日志中，因此這是一個包含新字段的新正則表達式：

• 主機名
• 簽名版本
• 密碼套件
• 認證類型
• 主機頭
• 轉換

還有一些其他的變化：

• 如果method+path 、 referrer或useragent值沒有被引號包圍，則 Yi 的最后一個 Regex 根本不匹配日志行，如果值為空（記錄為單個破折號），則通常是這種情況。
• 現在路徑的末尾附加了 HTTP 協議版本，因此我將其分離為一個新的協議值。

更新正則表達式

$pattern = '/(?P<owner>\S+) (?P<bucket>\S+) \[(?P<time>[^]]*)\] (?P<ip>\S+) (?P<requester>\S+) (?P<reqid>\S+) (?P<operation>\S+) (?P<key>\S+) (-|"-"|"(?P<method>[^ ]*) (?P<path>\S+) (?P<protocol>[^"]*)") (?P<status>\S+) (?P<error>\S+) (?P<bytes>\S+) (?P<size>\S+) (?P<totaltime>\S+) (?P<turnaround>\S+) (-|"(?P<referrer>[^"]*)") (-|"(?P<useragent>[^"]*)") (?P<version>\S+) (?P<hostid>\S+) (?P<sigversion>\S+) (?P<ciphersuite>\S+) (?P<authtype>\S+) (?P<hostheader>\S+) (?P<tlsversion>\S+)/';
preg_match($pattern, $string, $matches);

您可以替換空值（破折號）並從 $matches 數組中過濾掉重復的數字索引，如下所示：

$matches = array_map(
    function($val) { return $val === '-' ? '' : $val; },
    array_filter(
        $matches,
        function($key) { return !is_numeric($key); },
        ARRAY_FILTER_USE_KEY
    )
);