如何使用UFT从大型文本文件读取数据并在另一个字符串中写入包含字符串的行？

Question

I have problem at hand. I need to read a log file and if any line in the log file contains a specified log level, I need to write that line in a result file. I have written following code and it is working perfectly. But my problem is - the size of log file. In my project max size set for a log file is 100 MB. I tested this code with 2 MB - log file (15512 Lines) and it took ridiculously long time (about 1 hour 15 minutes). Also, I am not sure how will this behave with a large file. Do you have any other approach? Fast help will be really appreciated.

Option Explicit

Public Function chekLogFile(sLogFileName, sLogLevelToCheck, sResultFile)
    Dim oFSO, oFile, oResultFileObj, oResultFile
    Dim sFileContent
    Dim arrFileContent
    Dim iNumberOfLinesInFile, iCounter

    ' Open the result file to write
    Set oResultFileObj = CreateObject("Scripting.FileSystemObject")
    Set oResultFile = oResultFileObj.OpenTextFile(sResultFile,8)

    ' Read content from log file
    Set oFSO = CreateObject("Scripting.FileSystemObject")
    Set oFile = oFSO.OpenTextFile(sLogFileName,1)
    sFileContent = oFile.ReadAll()
    ' Create an array with content of each line as its elements
    arrFileContent = Split(sFileContent,vbcrlf)
    ' Get the number of lines
    iNumberOfLinesInFile = UBound(arrFileContent)

    ' If the line contails the log level, write the line in the result file
    ' The lines we are concereed about start as follows

    ' 20150823135921 :::: ERROR :: 
    ' 20150823135929 :::: WARNING :: 
    ' 20150823135930 :::: INFO :: 

    ' Please note: Any other occurrence of Either of the word except like above will not be counted.

    For iCounter = 0 To iNumberOfLinesInFile Step 1
         If Mid(arrFileContent(iCounter),21,Len(sLogLevelToCheck)) = sLogLevelToCheck Then
            oResultFile.WriteLine(arrFileContent(iCounter))
        End If
    Next

    ' Close the files
    oFile.Close
    oResultFile.Close

    ' Release the objects
    Set oResultFile = Nothing
    Set oFile = Nothing
    Set oFSO = Nothing
    Set oResultFileObj = Nothing
End Function


' Log level could be either ERROR OR WARNING OR INFO
Call chekLogFile("E:\UFTTrial\gmail.log", "ERROR", "E:\UFTTrial\ResultFile.txt")

Answer 1

A text file is a collection of strings. If you need to process it sequencially/line by line, slurping the file and then splitting the content into an array is a waste of time and memory. Use .ReadLine() instead.

Sample code applied to a 20 MB file took less than 2 min on my (slow) machine:

Option Explicit

Const ForAppending = 8
Const csSrcFile = "M:\lib\kurs0705\testdata\lines.txt"

Dim oFSO : Set oFSO = CreateObject("Scripting.FileSystemObject")

Dim dtStart : dtStart = Now()
checkLogFile csSrcFile, "This", "selected.txt"
Dim dtEnd   : dtEnd   = Now() - dtStart
WScript.Echo oFSO.GetFile(csSrcFile).Size / 10^6, "MB  ", FormatDateTime(dtEnd, vbShortTime)

Public Sub checkLogFile(sLogFileName, sLogLevelToCheck, sResultFile)
    Dim oInFile  : Set oInFile  = oFSO.OpenTextFile(sLogFileName)
    Dim oOutFile : Set oOutFile = oFSO.OpenTextFile(sResultFile, ForAppending, True)
    Do Until oInFile.AtEndOfStream
       Dim sLine : sLine = oInFile.ReadLine()
       If Mid(sLine, 1, Len(sLogLevelToCheck)) = sLogLevelToCheck Then
          oOutFile.WriteLine sLine
       End If
    Loop
    oInFile.Close
    oOutFile.Close
End Sub

output:

cscript readlog.vbs
20,888896 MB   00:01

Answer 2

The physical size of the file isn't the important aspect; the number of lines in the file is. The higher the line count, the longer it's going to take to use .ReadLine()

Ekkehard's answer is pretty much verbatim to what I was going to write. Keep in mind that a file with 2,000 lines and 200 characters/line will be read significantly faster than a 20,000 lines and 20 character/line file. How many lines are in the file you're trying to parse?