categorize a set of phrases into a set of similar phrases

Question

I have a few apps that generate textual tracing information (logs) to log files. The tracing information is the typical printf() style - ie there are a lot of log entries that are similar (same format argument to printf), but differ where the format string had parameters.

What would be an algorithm (url, books, articles, ...) that will allow me to analyze the log entries and categorize them into several bins/containers, where each bin has one associated format?
Essentially, what I would like is to transform the raw log entries into (formatA, arg0 ... argN) instances, where formatA is shared among many log entries. The formatA does not have to be the exact format used to generate the entry (even more so if this makes the algo simpler).

Most of the literature and web-info I found deals with exact matching, a max substring matching, or a k-difference (with k known/fixed ahead of time). Also, it focuses on matching a pair of (long) strings, or a single bin output (one match among all input). My case is somewhat different, since I have to discover what represents a (good-enough) match (generally a sequence of discontinuous strings), and then categorize each input entries to one of the discovered matches.

Lastly, I'm not looking for a perfect algorithm, but something simple/easy to maintain.

Thanks!

Answer 1

You can use the famous Bag of Words techniques tan help group texts by using a sparse matrix representing in each row the text (generally the text is stemmed with the porter stemmer for better results) After computing the bag of words, where you need to count the number of times each word appears in each piece of text and then calculating the totals of the matrix by rows and columns you calculate the tf-idf for each cell, in order to pay attention to the angular distance on the texts that is the one that best works. After doing all of this you can perform a clustering algorithm that groups the related pieces of texts, you can even extract from here the main keywords of the text. there is a program that does all this automatically that is called cluto , I strongly recommend it.