[英]tokenize sentence into words python
我想從不同的句子中提取信息,所以我使用 nltk 將每個句子划分為單詞,我正在使用以下代碼:
words=[]
for i in range(len(sentences)):
words.append(nltk.word_tokenize(sentences[i]))
words
它工作得很好,但我想要一些不同的東西..例如我有這句話: '[\'Jan 31 19:28:14 nginx: 10.0.0.0 - - [31/Jan/2019:19:28:14 +0100] "POST /test/itf/ HTTP/xx" 404 146 "-" "Mozilla/5.2 [en] (X11, U; OpenVAS-XX 9.2.7)"\']'
我想要"Mozilla/5.2 [en] (X11, U; OpenVAS-XX 9.2.7)"
是一個單詞,而不是分成幾個單詞。
更新:我想要這樣的東西:
[
'Jan',
'31',
'19:28:14',
'nginx',
'10.0.0.0',
'31/Jan/2019:19:28:14',
'+0100',
'POST',
'/test/itf/',
'HTTP/x.x',
'404',
'146',
'Mozilla/5.2 [en] (X11, U; OpenVAS-XX 9.2.7)']
任何想法使它成為可能? 先感謝您
您可以使用正則表達式import re
並解析日志行(這不是自然語言句子):
import re
sentences = ['[\'Jan 31 19:28:14 nginx: 10.0.0.0 - - [31/Jan/2019:19:28:14 +0100] "POST /test/itf/ HTTP/x.x" 404 146 "-" "Mozilla/5.2 [en] (X11, U; OpenVAS-XX 9.2.7)"\']']
rx = re.compile(r'\b(\w{3})\s+(\d{1,2})\s+(\d{2}:\d{2}:\d{2})\s+(\w+)\W+(\d{1,3}(?:\.\d{1,3}){3})(?:\s+\S+){2}\s+\[([^][\s]+)\s+([+\d]+)]\s+"([A-Z]+)\s+(\S+)\s+(\S+)"\s+(\d+)\s+(\d+)\s+\S+\s+"([^"]*)"')
words=[]
for sent in sentences:
m = rx.search(sent)
if m:
words.append(list(m.groups()))
else:
words.append(nltk.word_tokenize(sent))
print(words)
請參閱Python 演示。
output 看起來像
[['Jan', '31', '19:28:14', 'nginx', '10.0.0.0', '31/Jan/2019:19:28:14', '+0100', 'POST', '/test/itf/', 'HTTP/x.x', '404', '146', 'Mozilla/5.2 [en] (X11, U; OpenVAS-XX 9.2.7)']]
首先,您需要選擇使用 " 或 ' 因為兩者都很不尋常並且可能導致任何奇怪的行為。之后只是字符串格式:
s='"[\"Jan 31 19:28:14 nginx: 10.0.0.0 - - [31/Jan/2019:19:28:14 +0100] "POST /test/itf/ HTTP/x.x" 404 146 "-" "Mozilla/5.2 [en] (X11, U; OpenVAS-XX 9.2.7)"\"]" i want "Mozilla/5.2 [en] (X11, U; OpenVAS-XX 9.2.7)"'
words = s.split(' ') # break the sentence into spaces
# ['"["Jan', '31', '19:28:14', 'nginx:', '10.0.0.0', '-', '-', '[31/Jan/2019:19:28:14', '+0100]', '"POST', '/test/itf/', 'HTTP/x.x"', '404', '146', '"-"', '"Mozilla/5.2', '[en]', '(X11,', 'U;', 'OpenVAS-XX', '9.2.7)""]"', 'i', 'want', '"Mozilla/5.2', '[en]', '(X11,', 'U;', 'OpenVAS-XX', '9.2.7)"']
# then access your data list
words[0] # '"["Jan'
words[1] # '31'
words[2] # '19:28:14'
您可以使用parition()
和空格分隔符來做到這一點,並繼續對字符串進行分區,直到獲得所需的結果。 下面是解決方案。 不過我不得不說,這個解決方案對你提供的字符串格式是嚴格的。 它可能不是最好的方法,但會給您所需的 output。 尋找更優雅的解決方案的正則表達式。
s = '[\'Jan 31 19:28:14 nginx: 10.0.0.0 - - [31/Jan/2019:19:28:14 +0100] "POST /test/itf/ HTTP/x.x" 404 146 "-" "Mozilla/5.2 [en] (X11, U; OpenVAS-XX 9.2.7)"\']'
x = s.partition(" ")
s_list = []
s_list.append(x[0].replace("'", '').replace('[', ''))
x = x[2].partition(" ")
s_list.append(x[0])
x = x[2].partition(" ")
s_list.append(x[0])
x = x[2].partition(" ")
s_list.append(x[0].replace(":", ''))
x = x[2].partition(" ")
s_list.append(x[0])
x = x[2].partition(" ")
x = x[2].partition(" ")
x = x[2].partition(" ")
s_list.append(x[0].replace('[', ''))
x = x[2].partition(" ")
s_list.append(x[0].replace(']', ''))
x = x[2].partition(" ")
s_list.append(x[0])
x = x[2].partition(" ")
s_list.append(x[0])
x = x[2].partition(" ")
s_list.append(x[0].replace('"', ''))
x = x[2].partition(" ")
s_list.append(x[0])
x = x[2].partition(" ")
s_list.append(x[0])
x = x[2].partition(" ")
s_list.append(x[2].replace('"', '').replace(']', '').replace("'", ''))
print(s_list)
Output:
['Jan', '31', '19:28:14', 'nginx', '10.0.0.0', '31/Jan/2019:19:28:14', '+0100',
'"POST', '/test/itf/', 'HTTP/x.x', '404', '146', 'Mozilla/5.2 [en (X11, U; OpenV
AS-XX 9.2.7)']
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.