解析日志文件以提取查询

Question

I want to extract certain URL from a log file. But I only want to extract those queries that were ranked 1. or 2 . The log file contains a colum itemRank , giving the rank. So far I was able to extract certain URL by scanning through the text. But I do not know how to implement the condition that the URL is only clicked first or second.

For example , this is how part of the log file looks like:

(columns are ID,date, time, RANK, url)

763570 2006-03-06 14:09:48 2 http://something.com

763570 2006-03-06 14:09:48 3 http://something.com

Here I just want to extract the first query, because it was ranked 2.

---

This is my code so far:

public class Scanner {

    public static void main(String[] args) throws FileNotFoundException {


        File testFile = new File ("C:/Users/Zyaad/logs.txt");
        Scanner s = new Scanner(testFile);
        int count=0;

        String pattern="http://ontology.buffalo.edu";
        while(s.hasNextLine()){
            String line = s.nextLine();

            if (line.contains(pattern)){
                count++;

                System.out.println(count + ".query: " );
                System.out.println(line);
            } 

        }   System.out.println("url was clicked: "+ count + " times");

        s.close();

        }
}       

What can I do to just print out the 1. query? I tried regex like [\\t\\n\\b\\r\\f] [1,2]{1}[\\t\\n\\b\\r\\f] but this didn't work.

Answer 1

A simple (possibly simplistic) approach would be to:

1. Determine the number(s) (severity?) you're looking for
2. Determine a starting pattern for your URL

Example

// assume this is the file you're parsing so I don't have to repeat 
// the whole Scanner part here
String theFile = "763570 2006-03-06 14:09:48 2 http://something2.com\r\n" +
        "763570 2006-03-06 14:09:48 3 http://something3.com";
//                           | your starting digit of choice
//                           | | one white space
//                           | | | group 1 start
//                           | | | | partial protocol of the URL
//                           | | | |  | any character following in 1+ instances
//                           | | | |  | | end of group 1
//                           | | | |  | | 
Pattern p = Pattern.compile("2\\s(http.+)");
Matcher m = p.matcher(theFile);
while (m.find()) {
    // back-referencing group 1
    System.out.println(m.group(1));
}

Output

http://something2.com

Note

Parsing log files with regex is generally advised against.

You'd probably be better off long-term implementing your own parser and itemize tokens as properties of objects (1 per line I assume), then manipulate those as desired.

Answer 2

You can create a regex based on date & time pattern or you can simply start it from time pattern as well.

yyyy-MM-dd hh:mm:ss 1|2 

Date & Time pattern followed by 1 or 2

\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\s[1|2]\s

Time pattern followed by 1 or 2

\d{2}:\d{2}:\d{2}\s[1|2]\s

---

Sample code:

String[] str=new String[] { "763570 2006-03-06 14:09:48 2 http://something.com",
        "763570 2006-03-06 14:09:48 3 http://something.com" };

Pattern p = Pattern
          .compile("\\d{4}-\\d{2}-\\d{2}\\s\\d{2}:\\d{2}:\\d{2}\\s[1|2]\\s");
for (String s : str) {
    Matcher m = p.matcher(s);
    if (m.find()) {
        System.out.println(s.substring(m.end()));
    }
} 

Answer 3

You can find here some useful patterns . If it's possible to use other tools, i will suggest using logstash , an impressive tool for collecting and parsing log.

Answer 4

You can extract URLs ranked 1 or 2 like this:

/(?<=\s(?:1|2)\s).*$/

It will grab the last part of the line if the URL is preceded with either 1 or 2.

Answer 5

Try this:

public static void main(String[] args) throws FileNotFoundException {

    int count = 0;
    // create date pattern
    // source:https://github.com/elasticsearch/logstash/blob/master/patterns/grok-patterns
    String yearPattern = "(?>\\d\\d){1,2}";
    String monthNumPattern = "(?:0?[1-9]|1[0-2])";
    String monthDayPattern = "(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])";
    String hourPattern = "(?:2[0123]|[01]?[0-9])";
    String minutePattern = "(?:[0-5][0-9])";
    String secondPattern = "(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)";
    String datePattern = String.format("%s-%s-%s %s:%s:%s", yearPattern,
            monthNumPattern, monthDayPattern, hourPattern, minutePattern,
            secondPattern);

    // create url pattern
    // source: http://code.tutsplus.com/tutorials/8-regular-expressions-you-should-know--net-6149
    String urlPattern = "(https?://)?([\\da-z\\.-]+)\\.([a-z\\.]{2,6})([/\\w \\.-]*)*/?";
    Pattern pattern = Pattern.compile("(\\d+) (" + datePattern
            + ") (\\d+) (" + urlPattern + ")");
    String data = "763570 2006-03-06 14:09:48 3 http://something.com\n"
            + "763570 2006-03-06 14:09:48 2 http://something.com\n"
            + "763570 2006-03-06 14:09:48 1 http://something.com";
    ByteArrayInputStream is = new ByteArrayInputStream(data.getBytes());
    java.util.Scanner s = new java.util.Scanner(is);
    while (s.hasNextLine()) {
        String line = s.nextLine();
        Matcher matcher = pattern.matcher(line);
        if (matcher.matches()) {
            if (matcher.find(3)) {
                int rank = Integer.parseInt(matcher.group(3));
                if (rank == 1 || rank == 2) {
                    count++;
                }
            }
        }
    }
    System.out.println("url was clicked: " + count + " times");

    s.close();

}

this will output "url was clicked: 2 times" for file containing:

• 763570 2006-03-06 14:09:48 3 http://something.com
• 763570 2006-03-06 14:09:48 2 http://something.com
• 763570 2006-03-06 14:09:48 1 http://something.com