通过机器学习找到日常模式

Question

I have created a huge log of daily activity in the format [timestamp, location]. For example

[{1365650747255, 'san francisco'},
 {1365650743354, 'san francisco'},
 {1365650741349, 'san mateo'},
 {1365650756324, 'mountain view'},
 ...
 {1365650813354, 'menlo park'}]

What are the ways I can mine this information to find patterns like

• "On Sunday evenings, it's probable that I am near San Francisco"
• "On Monday afternoons it's probable that I am near Menlo Park"

The problem is that

• The dataset is huge.
• it looks impossible to judge the date/time/day by applying a function on the timestamp value (unless we decode the timestamp in to Date Time values).

Answer 1

I do not see your problem here. As it is a timestamp counting seconds from epoch you only have to apply the modulo operator with the value being the range of interest. If you train a classifier on that you should be able to predict every upcoming place. The main problem is not performance, as the learning is only done now and then, but how to update the learned dataset. As already stated you do not have to use machine learning for this however if you want to do it using machine learning this can basically be done using a k-nearest-neighbor on your 1d dataset.

[EDIT]: Mixed up languages but fixed it: A classifier is the algorithm which will do the statistical classification.

In machine learning and statistics, classification is the problem of identifying to which of a set of categories (sub-populations) a new observation belongs, on the basis of a training set of data containing observations (or instances) whose category membership is known. [ 1 ]

As I only have used sklearn to do such things the following is a minimalistic example of how you could use a k-nearest-neighbor classifier [ 2 ]. To be able to classify you have to change the strings into numbers, then train your classifier on the given test dataset and afterwards you are able to predict the location for a new given timestamp.

import numpy as np
from sklearn.neighbors import KNeighborsClassifier


data = [[1365650747255, 'san francisco'],
        [1365650743354, 'san francisco'],
        [1365650741349, 'san mateo'],
        [1365650756324, 'mountain view'],
        ...
        [1365650813354, 'menlo park']]

# Map location strings to integers and replace
location_mapping = {}
location_index = 0
for index, (time, location) in enumerate(data):
    if(not location_mapping.has_key(location)):
        location_mapping[location] = location_index
        location_index += 1

    data[index][1] = location_mapping[location]

inverse_location_mapping = {value:key for key, value in location_mapping.items()}

data = np.array(data)
week = 60 * 60 * 24 * 7

# Setup classifier
classifier = KNeighborsClassifier(n_neighbors=10)

# Train classifier on given data
classifier.fit(data[:, 0] % week, data[:, 1]) 

# Predict desired location
prediction = classifier.predict([[1365444444444 % week]]))
print(inverse_location_mapping[prediction])

[ 1 ] : http://en.wikipedia.org/wiki/Statistical_classification

[ 2 ] : http://scikit-learn.org/dev/modules/generated/sklearn.neighbors.KNeighborsClassifier.html

Answer 2

The performance is this solution depends on how granular your requirement for pattern recognition is. Lets assume your requirement is dividing the day into 4 parts : Morning,Noon,Evening,Night , lets call them time_slots

Now lets take a look at how big your daily activity log is, 1 year, 2 years , 3 years ?

lets assume it is 1 year.

So we have total of 365 * 4 = 1460 timeslots to monitor.

Now,create a simple map based on timestamps for each time_slot . Eg. It begins on T1 and ends on T2 ( where T1 and T2 are timestamps like 1365650813354 ).

Based on timestamp value in your log, it is easy to find its time_slot ie Evening of 28th January, or Morning of 30th January.

You will have to store time_slot vs place_i_was data in any suitable database with proper schema. That depends on kind of querying and analylsis you would want.

This way you will not need to run formulas on your dataset, and the predefined map/database lookup will serve your purpose.

Answer 3

Not sure these questions require machine learning, you can use regular statistics for that. Ie build a probability distribution plot, x - time of day, y - probability it is San Francisco. Calculate the probability of San Francisco if time is between a and b ...

---

This is how to load your data in pandas DataFrame:

from __future__ import print_function, division
import pandas as pd
import datetime

df = pd.read_csv("data.csv",
                 names=["timestamp","location"],
                 parse_dates=["timestamp"],
                 date_parser=lambda x:datetime.datetime.fromtimestamp(int(x) / 1000))
print(df.head())

Outputs:

                    timestamp          location
0  2013-04-11 04:25:47.255000   "san francisco"
1  2013-04-11 04:25:43.354000   "san francisco"
2  2013-04-11 04:25:41.349000       "san mateo"
3  2013-04-11 04:25:56.324000   "mountain view"
4  2013-04-11 04:26:53.354000      "menlo park"

Answer 4

Convert the timestamps into tokens: "sunday morning".

Then do association rule mining to obtain rules such as

night => home
sunday morning => running in the park

where you only keep those rules, where the desired locations occur on the right.

Answer 5

Firstly, convert the timestamp value to year-month-weekday. Replace the timestamp column by 3 columns corresponding to year, month and weekday.

Later you could simply group by certain range of values for dates and count the number of instances for each location.