[英]quicker way to detect n-grams in a string?
我在SO上發現了這個解決方案來檢測字符串中的n-gram :(這里: 句子中的N-gram生成 )
import java.util.*;
public class Test {
public static List<String> ngrams(int n, String str) {
List<String> ngrams = new ArrayList<String>();
String[] words = str.split(" ");
for (int i = 0; i < words.length - n + 1; i++)
ngrams.add(concat(words, i, i+n));
return ngrams;
}
public static String concat(String[] words, int start, int end) {
StringBuilder sb = new StringBuilder();
for (int i = start; i < end; i++)
sb.append((i > start ? " " : "") + words[i]);
return sb.toString();
}
public static void main(String[] args) {
for (int n = 1; n <= 3; n++) {
for (String ngram : ngrams(n, "This is my car."))
System.out.println(ngram);
System.out.println();
}
}
}
=>這段代碼占用了最長的處理時間(我的語料庫檢測到1克,2克,3克和4克為28秒:原始文本為4Mb),而其他操作則為毫秒(刪除停用詞等)
有沒有人知道Java中的解決方案會比上面提到的循環解決方案更快? (我在考慮多線程,使用集合,或者創造性的方法來分割字符串......?)謝謝!
你可以嘗試這樣的事情:
public class NGram {
private final int n;
private final String text;
private final int[] indexes;
private int index = -1;
private int found = 0;
public NGram(String text, int n) {
this.text = text;
this.n = n;
indexes = new int[n];
}
private boolean seek() {
if (index >= text.length()) {
return false;
}
push();
while(++index < text.length()) {
if (text.charAt(index) == ' ') {
found++;
if (found<n) {
push();
} else {
return true;
}
}
}
return true;
}
private void push() {
for (int i = 0; i < n-1; i++) {
indexes[i] = indexes[i+1];
}
indexes[n-1] = index+1;
}
private List<String> list() {
List<String> ngrams = new ArrayList<String>();
while (seek()) {
ngrams.add(get());
}
return ngrams;
}
private String get() {
return text.substring(indexes[0], index);
}
}
在大約5mb的文本上進行測試似乎比原始代碼快10倍。 這里的主要區別是正則表達式不用於拆分,並且ngram字符串不是通過串聯創建的。
更新:這是我在上述文本ngram 1-4上運行時得到的輸出。 我運行2GB內存來決定運行期間對GC的影響。 我跑了很多遍,以查看熱點編譯器的影響。
Loop 01 Code mine ngram 1 time 071ms ngrams 294121
Loop 01 Code orig ngram 1 time 534ms ngrams 294121
Loop 01 Code mine ngram 2 time 016ms ngrams 294120
Loop 01 Code orig ngram 2 time 360ms ngrams 294120
Loop 01 Code mine ngram 3 time 082ms ngrams 294119
Loop 01 Code orig ngram 3 time 319ms ngrams 294119
Loop 01 Code mine ngram 4 time 014ms ngrams 294118
Loop 01 Code orig ngram 4 time 439ms ngrams 294118
Loop 10 Code mine ngram 1 time 013ms ngrams 294121
Loop 10 Code orig ngram 1 time 268ms ngrams 294121
Loop 10 Code mine ngram 2 time 014ms ngrams 294120
Loop 10 Code orig ngram 2 time 323ms ngrams 294120
Loop 10 Code mine ngram 3 time 013ms ngrams 294119
Loop 10 Code orig ngram 3 time 412ms ngrams 294119
Loop 10 Code mine ngram 4 time 014ms ngrams 294118
Loop 10 Code orig ngram 4 time 423ms ngrams 294118
通過您提供的代碼運行大約5兆的Lorus Ipsum文本,通常需要大約7秒鍾多一點的時間才能檢測到1-4 n-gram。 我對代碼進行了重做,以制作出最長n-gram的列表,然后在此列表上進行迭代,從而生成連續更短的ngram的列表。 在測試中,相同的文本花費了大約2.6秒。 另外,它占用的內存少得多。
import java.util.*;
public class Test {
public static List<String> ngrams(int max, String val) {
List<String> out = new ArrayList<String>(1000);
String[] words = val.split(" ");
for (int i = 0; i < words.length - max + 1; i++) {
out.add(makeString(words, i, max));
}
return out;
}
public static String makeString(String[] words, int start, int length) {
StringBuilder tmp= new StringBuilder(100);
for (int i = start; i < start + length; i++) {
tmp.append(words[i]).append(" ");
}
return tmp.substring(0, tmp.length() - 1);
}
public static List<String> reduceNgrams(List<String> in, int size) {
if (1 < size) {
List<String> working = reduceByOne(in);
in.addAll(working);
for (int i = size -2 ; i > 0; i--) {
working = reduceByOne(working);
in.addAll(working);
}
}
return in;
}
public static List<String> reduceByOne(List<String> in) {
List<String> out = new ArrayList<String>(in.size());
int end;
for (String s : in) {
end = s.lastIndexOf(" ");
out.add(s.substring(0, -1 == end ? s.length() : end));
}
//the last one will always reduce twice - words 0, n-1 are in the loop this catches the words 1, n
String s = in.get(in.size() -1);
out.add(s.substring(s.indexOf(" ")+1));
return out;
}
public static void main(String[] args) {
long start;
start = System.currentTimeMillis();
List<String> ngrams = ngrams(3, "Your text goes here, actual mileage may vary");
reduceNgrams(ngrams, 3);
System.out.println(System.currentTimeMillis() - start);
}
}
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.