Python 正则表达式获取长路径中的文件名

Question

I need to analysis some log file and it look like below, I would like to retrieve 3 parts of data,

1. Time
2. part of directory, in this case, it would be ABC and DEF in input file.
3. file name in input file, ie 2C.013000000B.dat, 20100722B.TXT, 20100722D1-XYZ.TXT and 2C.250B in this case.

I use this regular expression, but it fail to get third part.

(\d\d:\d\d:\d\d).*(ABC|DEF).*\\(\d\w\.?\w\..*)\soutput.*

Any suggestion will be appreciated.

08:38:36   TestModule - [INFO]result success !! ftp_site=ftp.test.com file_dir=CPY input file=\root\level1\level2-ABC\2C.013000000B.dat output file=c:\local\project1\data\2C.013000000B.dat.ext
06:40:37   TestModule - [INFO]result success !! ftp_site=ftp.test.com file_dir=CPY input file=\root\level1\level2-ABC\20100722B.TXT output file=c:\local\project1\data\20100722B.TXT.ext
06:40:39   TestModule - [INFO]result success !! ftp_site=ftp.test.com file_dir=CPY input file=\root\level1\level2-DEF\20100722D1-XYZ.TXT output file=c:\local\project1\data\20100722D1-YFP.TXT.ext
06:40:42   TestModule - [INFO]result success !! ftp_site=ftp.test.com file_dir=CPY input file=\root\level1\level2-DEF\2C.250B output file=c:\local\project1\data\2C.250B.ext

BR

Edward

Answer 1

Using split is a good idea. If you really want a regex, I would do it like this:

(\d\d:\d\d:\d\d).*?input file=.*?(ABC|DEF)\\\\(.*?)\soutput

Test it here

Answer 2

Regular expressions are very good at solving problems like this one - ie parsing log file records. MarcoS's answer solves your immediate problem nicely. However, another approach is to write a (reusable) generalized function which decomposes a log file record into its various components and returns a match object containing all these parsed components. Once decomposed, tests can easily be applied to the component parts to check for various requirements (such as the input file path must end in ABC or DEF ). Here is a python script which has just such a function: decomposeLogEntry() and demonstrates how to use it to solve your problem at hand:

import re
def decomposeLogEntry(text):
    r""" Decompose log file entry into its various components.

    If text is a valid log entry, return regex match object of
    log entry components strings. Otherwise return None."""
    return re.match(r"""
        # Decompose log file entry into its various components.
        ^                            # Anchor to start of string
        (?P<time>\d\d:\d\d:\d\d)     # Capture: time
        \s+
        (?P<modname>\w+?)            # Capture module name
        \s-\s\[
        (?P<msgtype>[^]]+)           # Capture message type
        \]
        (?P<message>[^!]+)           # Capture message text
        !!\sftp_site=
        (?P<ftpsite>\S+?)            # Capture ftp URL
        \sfile_dir=
        (?P<filedir>\S+?)            # Capture file directory?
        \sinput\sfile=
        (?P<infile>                  # Capture input path and filename
          (?P<infilepath>\S+)\\      # Capture input file path
          (?P<infilename>[^\s\\]+)   # Capture input file filename
        )
        \soutput\sfile=
        (?P<outfile>                 # Capture input path and filename
          (?P<outfilepath>\S+)\\     # Capture output file path
          (?P<outfilename>[^\s\\]+)  # Capture output file filename
        )
        \s*                          # Optional whitespace at end.
        $                            # Anchor to end of string
        """, text, re.IGNORECASE | re.VERBOSE)

# Demonstrate decomposeLogEntry function. Print components of all log entries.
f=open("testdata.log")
mcnt = 0
for line in f:
    # Decompose this line into its components.
    m = decomposeLogEntry(line)
    if m:
        mcnt += 1
        print "Match number %d" % (mcnt)
        print "  Time:             %s" % m.group("time")
        print "  Module name:      %s" % m.group("modname")
        print "  Message type:     %s" % m.group("time")
        print "  Message:          %s" % m.group("message")
        print "  FTP site URL:     %s" % m.group("ftpsite")
        print "  Input file:       %s" % m.group("infile")
        print "  Input file path:  %s" % m.group("infilepath")
        print "  Input file name:  %s" % m.group("infilename")
        print "  Output file:      %s" % m.group("outfile")
        print "  Output file path: %s" % m.group("outfilepath")
        print "  Output file name: %s" % m.group("outfilename")
        print "\n",
f.close()

# Next pick out only the desired data.
f=open("testdata.log")
mcnt = 0
matches = []
for line in f:
    # Decompose this line into its components.
    m = decomposeLogEntry(line)
    if m:
        # See if this record meets desired requirements
        if re.search(r"ABC$|DEF$", m.group("infilepath")):
            matches.append(line)
f.close()
print "There were %d matching records" % len(matches)

This function not only picks out the various parts you are interested in, it also validates the input and rejects badly formatted records. Once written and debugged, this function can be reused by other programs which need to analyze the log files for other requirements.

Here is the output from the script when applied to your test data:

r"""
Match number 1
  Time:             08:38:36
  Module name:      TestModule
  Message type:     08:38:36
  Message:          result success
  FTP site URL:     ftp.test.com
  Input file:       \root\level1\level2-ABC\2C.013000000B.dat
  Input file path:  \root\level1\level2-ABC
  Input file name:  2C.013000000B.dat
  Output file:      c:\local\project1\data\2C.013000000B.dat.ext
  Output file path: c:\local\project1\data
  Output file name: 2C.013000000B.dat.ext

Match number 2
  Time:             06:40:37
  Module name:      TestModule
  Message type:     06:40:37
  Message:          result success
  FTP site URL:     ftp.test.com
  Input file:       \root\level1\level2-ABC\20100722B.TXT
  Input file path:  \root\level1\level2-ABC
  Input file name:  20100722B.TXT
  Output file:      c:\local\project1\data\20100722B.TXT.ext
  Output file path: c:\local\project1\data
  Output file name: 20100722B.TXT.ext

Match number 3
  Time:             06:40:39
  Module name:      TestModule
  Message type:     06:40:39
  Message:          result success
  FTP site URL:     ftp.test.com
  Input file:       \root\level1\level2-DEF\20100722D1-XYZ.TXT
  Input file path:  \root\level1\level2-DEF
  Input file name:  20100722D1-XYZ.TXT
  Output file:      c:\local\project1\data\20100722D1-YFP.TXT.ext
  Output file path: c:\local\project1\data
  Output file name: 20100722D1-YFP.TXT.ext

Match number 4
  Time:             06:40:42
  Module name:      TestModule
  Message type:     06:40:42
  Message:          result success
  FTP site URL:     ftp.test.com
  Input file:       \root\level1\level2-DEF\2C.250B
  Input file path:  \root\level1\level2-DEF
  Input file name:  2C.250B
  Output file:      c:\local\project1\data\2C.250B.ext
  Output file path: c:\local\project1\data
  Output file name: 2C.250B.ext

There were 4 matching records
"""

Answer 3

If you use a regex tool, it will make your life a lot easier for troubleshooting regex. Try this free one - there are probably better ones, but this works great. You can paste your log file there, and try your regex a little bit at a time, and it will highlight matches in real time.

Answer 4

Why regex?

Consider using split to get all words. This will give you the timestamp directly. Then go through all other words, check if there's a = in them, split them again in this case and there you have your paths and other parameters nicely. Standard Python path handling ( os.path ) will aid you at getting folder and file names.

Of course this approach fails if your path names may contain spaces, but otherwise it is definitely worth consideration.

Answer 5

You can do it simply by normal string processing

f=open("file")
for line in f:
    date,b = line.split("input")
    print "time: " , date.split()[0]
    input_path = b.split("output")[0]
    tokens=input_path.split("\\")
    filename=tokens[-1]
    directory=tokens[-2].split("-")[-1]
    print filename, directory
f.close()

Answer 6

This worked for your examples:

r'(\d\d:\d\d:\d\d).*(ABC|DEF).*?([^\\]*)\soutput.*'

Although a well written regular expression is appropriate here, I would have approached this differently. Most specifically, os.path.split is designed to separate filenames from base paths, and deals with all the corner cases that this regular expression ignores.