[英]Java - Find string pattern in a huge file failing
Working on a Spring Batch project.正在处理 Spring 批处理项目。
As an input to the method, we are passing a HUUGE file of approx 3 million lines.作为该方法的输入,我们传递了一个大约 300 万行的巨大文件。 We need to scan this file and filter the lines which don't have the SQL codes mentioned in a
whitelistedVals list
.我们需要扫描此文件并过滤掉
whitelistedVals list
没有提到的 SQL 代码的行。 But my code is taking too much time to read such a huge file.但是我的代码花费了太多时间来读取这么大的文件。 It is working fine for other small sized files though.
不过,它对其他小文件也能正常工作。
public class MyClass {
private static final List<String> whitelistedVals = new ArrayList<>();
static {
whitelistedVals.add("SQL123N");
whitelistedVals.add("SQL2340W");
whitelistedVals.add("SQL3459W");
}
public String getSqlError(String inputFile) {
Pattern r = Pattern.compile("(SQL\\d+[A-Z]*)(?s)(.+?)(\\n\\n|\\n\\Z)");
Matcher m = r.matcher(inputFile);
String error = "";
while (m.find()) {
String errorCode = m.group(1);
String errorInGroup = errorCode + m.group(2).toUpperCase();
boolean errorsFound = whitelistedVals
.stream()
.noneMatch(x -> x.equalsIgnoreCase(errorCode));
if (errorsFound) {
error += errorInGroup;
}
}
return error;
}
}
Any suggestions on how this can be handled to speed up the process?关于如何处理此问题以加快流程的任何建议?
Using a StringBuilder instead of concat (+=) worked like a charm.使用 StringBuilder 而不是 concat (+=) 就像一个魅力。
It seems, the entire file is read and then its contents is supplied to getSqlError
method while it could be better to scan the file using double line-feeds \n
as a delimiter.看起来,整个文件都被读取,然后将其内容提供给
getSqlError
方法,而使用双换行符\n
作为分隔符扫描文件可能会更好。
Also, whitelistedVals
are streamed for each match, although they could be integrated into the pattern.此外,
whitelistedVals
会针对每场比赛进行流式传输,尽管它们可以集成到模式中。
So the method could look as follows:所以该方法可能如下所示:
List<String> whitelistedVals = Arrays.asList("123N", "2340W", "3459W");
public static String getSqlError(String inputFile) throws Exception {
Scanner scan = new Scanner(new File(inputFile))
.useDelimiter("\\n\\n|\\n\\Z");
final Spliterator<String> splt = Spliterators.spliterator(scan, Long.MAX_VALUE, Spliterator.ORDERED | Spliterator.NONNULL);
Pattern r = Pattern.compile("(SQL(?!("
+ whitelistedVals.stream().collect(Collectors.joining("|"))
+ ")\\b)(\\d+[A-Z]*))(?s)(.+)");
return StreamSupport.stream(splt, false).onClose(scan::close)
.flatMap(s -> r.matcher(s)
.results() // Stream<MatchResult>
.map(mr -> mr.group(1) + mr.group(4)) // errorCode + error
.map(String::toUpperCase)
) // Stream<String>
.collect(Collectors.joining("\n"));
}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.