根据data.table中各组之间的匹配返回一个变量

Question

I'm new to data.table and don't fully understand it. Suppose I have the following table of ngrams:

require(data.table)
DT<-data.table(
  ngram=c("how","how are","how are you","how are you doing"),
  Freq=c(15000,1500,150,15),
  n=c(1,2,3,4),
  w1=c(37,37,37,37),
  w2=c(NA,13,13,13),
  w3=c(NA,NA,7,7),
  w4=c(NA,NA,NA,95)
)

> DT
               ngram  Freq n w1 w2 w3 w4
1:               how 15000 1 37 NA NA NA
2:           how are  1500 2 37 13 NA NA
3:       how are you   150 3 37 13  7 NA
4: how are you doing    15 4 37 13  7 95

Where n denotes the type of ngram (eg 1=unigram, 2=bigram, etc), w1 through w4 are integer indexes of the words in each ngram, and Freq is the count of ngram occurrence in the data.

How would I get Freq of one ngram based on a match of one word in that ngram with one word in another ngram, eg for the bigram (n=2) 'how are' how would I get Freq of unigram 'how' by matching w1 of 'how are' with w1 of 'how'? Or, for the trigram 'how are you', how would I get Freq of bigram 'how are' by matching w1+w2 of 'how are you' with w1+w2 of 'how are'?

I've tried, for example:

DT[n==2,B:=Freq[match(w1[n==1],w1[n==2])]]

and

DT[n==2,B:=Freq[which(w1[n==1]==w1[n==2])]]

But get only NAs:

               ngram  Freq n w1 w2 w3 w4  B
1:               how 15000 1 37 NA NA NA NA
2:           how are  1500 2 37 13 NA NA NA
3:       how are you   150 3 37 13  7 NA NA
4: how are you doing    15 4 37 13  7 95 NA

I would like to get:

               ngram  Freq n w1 w2 w3 w4     B
1:               how 15000 1 37 NA NA NA    NA
2:           how are  1500 2 37 13 NA NA 15000
3:       how are you   150 3 37 13  7 NA  1500
4: how are you doing    15 4 37 13  7 95   150

Any help greatly appreciated!

Answer 1

You can go through row by row, find the 'w' columns to be used as joining keys and then perform the join on these w columns with rows having smaller ngrams than the current row:

DT[, B := 
    {
        k <- as.integer(.BY) - 1L
        if (k > 0) {
            nm <- head(grep("^w", names(.SD)[!is.na(.SD)], value=TRUE), k)
            DT[n < .BY][.SD, x.Freq, on=nm]
        } else NA_real_
    },
    by=.(n)]

output:

               ngram  Freq n w1 w2 w3 w4     B
1:               how 15000 1 37 NA NA NA    NA
2:           how are  1500 2 37 13 NA NA 15000
3:       how are you   150 3 37 13  7 NA  1500
4: how are you doing    15 4 37 13  7 95   150

---

trimming code after Frank's comments:

DT[, B := 
    {
        if (n > 1L) {
            nm <- head(grep("^w", names(.SD)[!is.na(.SD)], value=TRUE), n-1L)
            DT[n==.BY$n-1L][.SD, x.Freq, on=nm]
        }
    },
    by=.(n)]

Answer 2

A variation on chinsoon's answer, overwriting the nth word to NA before joining:

wcols = paste0("w", 1:4)    
DT[, v := 
  DT[n == .BY$n - 1L][replace(.SD, .BY$n, NA_real_), on=wcols, x.Freq]
, by=n, .SDcols=wcols]

That this approach, while more concise, is probably less efficient, since I am joining on all columns instead of just n-1 .

Answer 3

I keyed n, made B a subset of DT, and reversed the order of the match:

setkey(DT,n)
DT[.(2),B:=DT[,Freq[match(w1[n==2L],w1[n==1L],nomatch=NA)]]]

> DT
               ngram  Freq n w1 w2 w3 w4     B
1:               how 15000 1 37 NA NA NA    NA
2:           how are  1500 2 37 13 NA NA 15000
3:       how are you   150 3 37 13  7 NA  1500
4: how are you doing    15 4 37 13  7 95   150

Works quickly on large data set.