使用awk / sed命令如何從日志文件中過濾字段

Question

worker1.log內容如下

---

[2019-06-28 10:18:22,927: INFO/Worker-1] deleted files: []
[2019-06-28 10:18:26,382: INFO/MainProcess] Task phim_backoffice.persist.poperator_gen[5379f7e3-a2b8-4f25-989e-5f1f1da02965] succeeded in 19.2708234414s: None
[2019-07-05 09:18:31,203: INFO/MainProcess] Connected to amqp://guest:**@sb.phim.isyntax.net:5672//
[2019-07-05 09:18:31,225: INFO/MainProcess] mingle: searching for neighbors
[2019-07-05 09:18:32,327: INFO/MainProcess] mingle: all alone
[2019-07-05 09:18:32,372: WARNING/MainProcess] worker1@vigilant.idm.isyntax.net ready.
[2019-07-05 09:19:18,304: INFO/MainProcess] Received task: phim_backoffice.routers.do_discovery[2fb1181f-4190-4734-baf6-54ce81e8b00f]
[2019-07-05 09:19:18,340: INFO/MainProcess] Task phim_backoffice.routers.do_discovery[2fb1181f-4190-4734-baf6-54ce81e8b00f] succeeded in 0.0327974446118s: None
[2019-07-05 09:19:18,342: INFO/MainProcess] Received task: phim_backoffice.fact.discovery_collect[2e5a26ac-c74e-4f2d-9f1f-846a50fcaffd]
[2019-07-05 09:19:18,493: INFO/MainProcess] Received task: phim_backoffice.datastore.mongodb[d954789e-295c-4ebc-aa19-c37146fa71c9]
[2019-07-05 09:19:18,495: INFO/MainProcess] Received task: phim_backoffice.datastore.mongodb[36bf7a61-7c55-4c15-8d38-1401374f6c96]
[2019-07-05 09:19:18,501: INFO/MainProcess] Received task: phim_backoffice.datastore.mongodb[e3d76ba2-f5da-47e7-8bbf-ab985e46f50c]
[2019-07-05 09:19:18,503: INFO/MainProcess] Received task: phim_backoffice.datastore.mongodb[96859921-5f2b-411a-ac36-2de83fb41479]
[2019-07-05 09:19:18,555: INFO/MainProcess] Task phim_backoffice.fact.discovery_collect[2e5a26ac-c74e-4f2d-9f1f-846a50fcaffd] succeeded in 0.210998585448s: None
[2019-07-05 09:19:18,557: INFO/MainProcess] Received task: phim_backoffice.datastore.mongodb[40a5a263-9193-4358-93a2-51dc4457ade8]
[2019-07-05 09:19:18,559: INFO/MainProcess] Task phim_backoffice.datastore.mongodb[d954789e-295c-4ebc-aa19-c37146fa71c9] succeeded in 0.00273974426091s: None
[2019-07-05 09:19:18,561: INFO/MainProcess] Received task: phim_backoffice.datastore.mongodb[df36f9e6-e636-465d-b26f-6aa698b68b33]
[2019-07-05 09:19:18,562: INFO/MainProcess] Task phim_backoffice.datastore.mongodb[36bf7a61-7c55-4c15-8d38-1401374f6c96] succeeded in 0.00246330350637s: None
[2019-07-05 09:19:18,564: INFO/MainProcess] Received task: phim_backoffice.datastore.mongodb[12f3fb74-186b-4649-9272-3aadb73ec0e9]
[2019-07-05 09:19:18,565: INFO/MainProcess] Task phim_backoffice.datastore.mongodb[e3d76ba2-f5da-47e7-8bbf-ab985e46f50c] succeeded in 0.00229033827782s: None
[2019-07-05 09:19:18,567: INFO/MainProcess] Received task: phim_backoffice.datastore.mongodb[8e35d073-d3bf-4bcb-9af8-54e29c9e2ba5]
[2019-07-05 09:19:18,568: INFO/MainProcess] Task phim_backoffice.datastore.mongodb[96859921-5f2b-411a-ac36-2de83fb41479] succeeded in 0.00237063132226s: None
[2019-07-05 09:19:18,638: INFO/MainProcess] Task phim_backoffice.datastore.mongodb[40a5a263-9193-4358-93a2-51dc4457ade8] succeeded in 0.00242230296135s: None
[2019-07-05 09:19:18,641: INFO/MainProcess] Task phim_backoffice.datastore.mongodb[df36f9e6-e636-465d-b26f-6aa698b68b33] succeeded in 0.0020351447165s: None
[2019-07-05 09:19:18,644: INFO/MainProcess] Task phim_backoffice.datastore.mongodb[12f3fb74-186b-4649-9272-3aadb73ec0e9] succeeded in 0.00239575095475s: None
[2019-07-05 09:19:18,647: INFO/MainProcess] Task phim_backoffice.datastore.mongodb[8e35d073-d3bf-4bcb-9af8-54e29c9e2ba5] succeeded in 0.00195684097707s: None
[2019-07-05 09:19:18,830: INFO/MainProcess] Received task: phim_backoffice.datastore.mongodb[f1c129ab-8cb4-452b-912b-dd751d0fb868]
[2019-07-05 09:19:18,831: INFO/MainProcess] Received task: phim_backoffice.datastore.mongodb[c09d7d87-0aff-49d4-96ee-eb1441674382]
[2019-07-05 09:19:18,832: INFO/MainProcess] Received task: phim_backoffice.datastore.mongodb[bdd1270e-9158-4f79-91a1-459bf17f63f7]
[2019-07-05 09:19:18,833: INFO/MainProcess] Received task: phim_backoffice.datastore.mongodb[f08ec613-d8b3-48c3-b786-ab5b5625d70d]
[2019-07-05 09:19:18,835: INFO/MainProcess] Received task: phim_backoffice.datastore.mongodb[f07d2c4b-aa9d-4962-b70c-71ebc38f9eee]
[2019-07-05 09:19:18,837: INFO/MainProcess] Task phim_backoffice.datastore.mongodb[f1c129ab-8cb4-452b-912b-dd751d0fb868] succeeded in 5.00330241024494s: None
[2019-07-05 09:19:18,839: INFO/MainProcess] Received task: phim_backoffice.datastore.mongodb[0a56c404-973e-43cb-911d-c302b99cc161]
[2019-07-05 09:19:18,841: INFO/MainProcess] Task phim_backoffice.datastore.mongodb[c09d7d87-0aff-49d4-96ee-eb1441674382] succeeded in 0.00253757275641s: None
[2019-07-05 09:19:18,843: INFO/MainProcess] Received task: phim_backoffice.datastore.mongodb[222162f6-7525-4355-8556-0b944373607b]
[2019-07-05 09:19:18,844: INFO/MainProcess] Task phim_backoffice.datastore.mongodb[bdd1270e-9158-4f79-91a1-459bf17f63f7] succeeded in 0.00211428850889s: None
[2019-07-05 09:19:18,846: INFO/MainProcess] Received task: phim_backoffice.datastore.mongodb[5fb1ad41-74a5-4987-91dc-e4538c4e01eb]
[2019-07-05 09:19:18,847: INFO/MainProcess] Task phim_backoffice.datastore.mongodb[f08ec613-d8b3-48c3-b786-ab5b5625d70d] succeeded in 0.00255416892469s: None
[2019-07-05 09:19:18,849: INFO/MainProcess] Received task: phim_backoffice.datastore.mongodb[e5e94eb2-bc5e-4851-bd91-2fcf070c1740]
[2019-07-05 09:19:18,850: INFO/MainProcess] Task phim_backoffice.datastore.mongodb[f07d2c4b-aa9d-4962-b70c-71ebc38f9eee] succeeded in 12.00233167037368s: None
[2019-07-05 09:19:18,852: INFO/MainProcess] Received task: phim_backoffice.fact.discovery_purge_missing[bc1aaff5-cc0a-4c1f-b625-adeb3553beab]
[2019-07-05 09:19:18,853: INFO/MainProcess] Task phim_backoffice.datastore.mongodb[0a56c404-973e-43cb-911d-c302b99cc161] succeeded in 2.00230784527957s: None
[2019-07-05 09:19:18,855: INFO/MainProcess] Received task: phim_backoffice.persist.generate_from_facts[04b82ebe-9982-4d20-9a3f-8438143b0889] eta:[2019-07-05 09:19:24.553480+00:00]
[2019-07-05 09:19:18,857: INFO/MainProcess] Task phim_backoffice.datastore.mongodb[222162f6-7525-4355-8556-0b944373607b] succeeded in 0.0029069762677s: None
[2019-07-05 09:19:18,859: INFO/MainProcess] Task phim_backoffice.datastore.mongodb[5fb1ad41-74a5-4987-91dc-e4538c4e01eb] succeeded in 5.00210771337152s: None
[2019-07-05 09:19:18,863: INFO/MainProcess] Task phim_backoffice.datastore.mongodb[e5e94eb2-bc5e-4851-bd91-2fcf070c1740] succeeded in 10.00259610638022s: None

我需要過濾成功時間超過5秒的條目

預期的輸出格式如下

phim_backoffice.persist.poperator_gen : 19.2708234414
phim_backoffice.datastore.mongodb_gen : 5.00330241024494
phim_backoffice.datastore.mongodb_gen : 12.00233167037368
phim_backoffice.datastore.mongodb_gen : 5.00210771337152
phim_backoffice.datastore.mongodb_gen : 10.00259610638022

我如何使用awk或sed或linux中的任何其他命令來實現這一點。

我嘗試使用下面的腳本，但由於該字段中有一個“ s” （例如-10.00259610638022s） ，因此其無法正常工作

awk '/succeeded/ { if ($(NF-1) > 5) print $1,":",$2, ":", $5, ":", $(NF-1);}' /tmp/worker1.log

Answer 1

grep succeeded worker1.log | sed 's/.*Task \\(.*\\)\\[.*succeeded in \\(.*\\)s:.*/\\1 : \\2/' | awk '{ if ($3 > 5) print }'

結果是：

phim_backoffice.datastore.mongodb : 5.00330241024494
phim_backoffice.datastore.mongodb : 12.00233167037368
phim_backoffice.datastore.mongodb : 5.00210771337152
phim_backoffice.datastore.mongodb : 10.00259610638022```

Answer 2

嘗試這個：

sed -r -e "s/\[[^]]*\] ?//g" -e "s/s://" worker1.log | sed -rn "/^Task/ s/ in / : /p" | awk '{ if ($4 > 5.0) { print $2, $3, $4; } }'

Answer 3

這是一個使用FPAT內部變量提取所需數據的簡單awk腳本。

一線

awk 'BEGIN{FPAT="succeeded in [^:]*"}{a=substr($1,14)+0;if(a>5)print}' input.txt

或script.awk

BEGIN{FPAT="succeeded in [^:]*"} # extract into $1 field the text matched to FPAT RegExp
{
    a = substr($1,14) + 0; # extract text from $1 field and convert to number
    if (a > 5) print; # print the whole line if a > 5
}

運行：

awk -f script.awk input.txt