光流和手指跟踪

Question

I am using opencv to implement finger tracking system And also use calcOpticalFlowPyrLK(pGmask,nGmask,fingers,track,status,err); to perform a LK tracker.

The concept I am not clear, after I implement the LK tracker, how should I detect the movement of fingers? Also, the tracker get the last frame and current frame, how to detect a series of action or continuous gesture like within 5 frames?

Answer 1

The 4th parameter of calcOpticalFlowPyrLK (here track ) will contain the calculated new positions of input features in the second image (here nGmask ).

In the simple case, you can estimate the centroid separately of fingers and track where you can infer to the movement. Making decision can be done from the direction and magnitude of the vector pointing from fingers ' centroid to track 's centroid.

Furthermore, complex movements can be considered as time series, because movements are consisting of some successive measurements made over a time interval. These measurements could be the direction and magnitude of the vector mentioned above. So any movement can be represented as below:

("label of movement", time_series), where
time_series = {(d1, m1), (d2, m2), ..., (dn, mn)}, where
di is direction and mi is magnitude of the ith vector (i=1..n)

So the time-series consists of n * 2 measurements (sampling n times), that's the only question how to recognize movements?

If you have prior information about the movement, ie you know how to perform a circular movement, write an a letter etc. then the question can be reduced to: how to align time series to themselves?

Here comes the well known Dynamic Time Warping (DTW). It can be also considered as a generative model, but it is used between pairs of sequences. DTW is an algorithm for measuring similarity between two temporal sequences which may vary in time or speed (such in our case).

In general, DTW calculates an optimal match between two given time series with certain restrictions. The sequences are warped non-linearly in the time dimension to determine a measure of their similarity independent of certain non-linear variations in the time dimension.