如何解析特定的数据文件并对其内容进行聚类？ Java的

Question

I have a file just like following,

150 event4
160 event4
160 event0
170 event4
175 event4
180 event4
190 event4
192 event3
195 event4
----------
----------

The first column is the time in milisecond the corresponding event actually occurred. so event4 occured in 150 milisecond.

I have following tasks to do,

1. Iterate through the lines one by one.

2. If there is a gap between consecutive events less than 80 millisecond they are a sequence of a single activity.

for example

100 event4
120 event5 
140 event6
200 event4

all of them have got consecutive difference not more than 80 millisecond. If there is a difference more than 80 millisecond that means current sequence ended and new sequence started. My goal is to cluster the sequences. And in different clusters report the number of particular events. So, in the following example in cluster 1 event 4 occurred 4 times, event 5 1 and event 6 1 time. in the second cluster event 4 3 times and event5 1 time.

100 event4
120 event5 
140 event6
200 event4

300 event4
320 event4 
340 event4
400 event5

What I am doing now is that,

1. I make a list of strings. I parse the file, and measure the gap between lines if it is less than 80 millisecond I add them to the list.
2. when I found an event with more than 80 millisecond gap I stop adding and create a new list for next sequence.
3. after having all the sequence in different lists i then traverse through the lists to measure the number of particular events.

I dont know this is an efficient approach or not. I have certain problems.

• I do not know how many cluster of sequences over there, so the number of lists i want to store particular clusters is not fixed.
• The event names are not fixed. it can be event1 to event100 or event 1 to 45. So, number of variables used to store event numbers is not fixed too.

So, do u guys have any more good ideas?

Answer 1

It's not what one would call "clustering" in science, but just grouping or aggregation. You aggregate events unless they are separated by too much time.

As for the approach, you are pursuing the canonical approach. You can't do better than linear unless your data is already in a complex data base index. As long as it is a text file, there is no way except to read it linearly.

As for the data structures, there is nothing wrong with organizing it as an ArrayList<ArrayList<String>> or ArrayList<HashMap<String, Integer>> , as the event IDs are strings. The memory requirements should be moderate and scale up to a Gigabyte. If you are running into memory problems, try maintaining a HashSet<String> to keep only one copy of each event string, and convert the time to a numerical data type. You should then be able to load several GB when you have few enough events.

Actually I don't see any major challenge here.