如何從一個data.table中選擇行以應用於另一data.table？

Question

我有兩個data.tables df （21 MIO行）和tmp （500k行）

df具有三列，將原始專利（ origpat ）與參考專利（ refpat ）鏈接在一起，並將原始分類（ mainprim ）綁定到origpat 。

在下面的30行中顯示。 每個origpat ， refpat 對都是唯一的，但每個origpat出現1至300次，每個refpat之間1至3,100次

dput(df[1:30,-3])
structure(list(origpat = c(4247592, 4247592, 4247592, 4247592, 
4247592, 4247592, 4247592, 4247592, 4247592, 4247592, 4247592, 
4247592, 4247592, 4247592, 4247592, 4247592, 4247592, 4247592, 
4247592, 4247592, 4247592, 4247592, 4247592, 4247592, 4247592, 
4247592, 4247592, 4247592, 4247592, 4247592), ref.pat = c(4318978, 
4436368, 4358181, 4478622, 4312654, 4293439, 4286061, 4363648, 
4406517, 4478623, 4277285, 4375743, 4470520, 4328022, 4248614, 
4297139, 4296607, 4296608, 4395271, 4321141, 4294190, 4431420, 
4322467, 4285730, 4393138, 4246034, 4251278, 4339174, 4277322, 
4290586), mainprim = c("442", "442", "442", "442", "442", "442", 
"442", "442", "442", "442", "442", "442", "442", "442", "442", 
"442", "442", "442", "442", "442", "442", "442", "442", "442", 
"442", "442", "442", "442", "442", "442")), .Names = c("origpat", 
"ref.pat", "mainprim"), row.names = c(NA, 30L), class = c("data.table", 
"data.frame"))

tmp包含一個專利pnum列表，其主要分類為prim 。 df中的所有origpat和refpat都是tmp中的pnum （專利號）。 作為示例數據，我給出了tmp data.table的選擇，其中包含與通過dput選定的df變量有關的所有信息

dput(tmp)
structure(list(pnum = c("4318978", "4318978", "4318978", "4318978", 
"4318978", "4318978", "4318978", "4318978", "4436368", "4436368", 
"4436368", "4436368", "4358181", "4358181", "4358181", "4358181", 
"4478622", "4312654", "4312654", "4312654", "4312654", "4312654", 
"4312654", "4293439", "4293439", "4293439", "4293439", "4293439", 
"4293439", "4293439", "4293439", "4293439", "4293439", "4293439", 
"4293439", "4293439", "4286061", "4286061", "4286061", "4286061", 
"4286061", "4286061", "4286061", "4286061", "4363648", "4363648", 
"4363648", "4406517", "4478623", "4478623", "4277285", "4375743", 
"4375743", "4375743", "4375743", "4470520", "4470520", "4470520", 
"4328022", "4328022", "4248614", "4248614", "4248614", "4248614", 
"4248614", "4248614", "4297139", "4297139", "4297139", "4297139", 
"4297139", "4296607", "4296607", "4296607", "4296607", "4296607", 
"4296607", "4296608", "4296608", "4296608", "4296608", "4296608", 
"4395271", "4395271", "4395271", "4321141", "4321141", "4321141", 
"4321141", "4294190", "4294190", "4294190", "4294190", "4294190", 
"4294190", "4431420", "4431420", "4431420", "4431420", "4431420", 
"4431420", "4322467", "4322467", "4322467", "4322467", "4322467", 
"4322467", "4322467", "4322467", "4322467", "4322467", "4285730", 
"4285730", "4393138", "4393138", "4393138", "4393138", "4393138", 
"4393138", "4393138", "4246034", "4246034", "4246034", "4246034", 
"4251278", "4251278", "4251278", "4339174", "4339174", "4339174", 
"4339174", "4277322", "4277322", "4290586", "4290586", "4290586", 
"4290586", "4290586", "4247592", "4247592", "4247592", "4247592", 
"4247592", "4247592", "4247592", "4247592", "4247592"), prim = c("430", 
"430", "430", "430", "430", "430", "430", "430", "340", "385", 
"385", "385", "385", "385", "65", "65", "65", "118", "427", "65", 
"65", "65", "65", "106", "106", "106", "501", "501", "501", "501", 
"501", "516", "516", "516", "516", "516", "435", "435", "435", 
"435", "435", "435", "435", "435", "156", "428", "65", "385", 
"65", "65", "501", "422", "53", "53", "53", "222", "422", "604", 
"65", "65", "385", "385", "65", "65", "65", "65", "106", "106", 
"501", "501", "501", "252", "423", "423", "501", "505", "62", 
"423", "501", "501", "505", "62", "65", "65", "65", "210", "210", 
"210", "435", "118", "118", "118", "118", "118", "118", "106", 
"433", "433", "433", "433", "501", "156", "427", "427", "428", 
"428", "428", "428", "428", "428", "428", "501", "501", "426", 
"426", "426", "435", "435", "435", "435", "428", "501", "501", 
"501", "501", "501", "65", "385", "385", "385", "65", "204", 
"204", "204", "266", "266", "432", "73", "427", "427", "428", 
"442", "442", "442", "442", "8", "8")), .Names = c("pnum", "prim"
), class = c("data.table", "data.frame"), row.names = c(NA, -147L
), .internal.selfref = <pointer: 0x0000000000100788>)

現在，我想比較mainprim （鏈接到origpat ）和鏈接到refpat的不同prim變量。

下面的代碼有效，但是速度太慢。

library(data.table)
df <- data.table(df) ; setkey(df, refpat, origpat) 
refs <- unique(df$refpat) # Capture all unique refpat in df (71,000 in entire data.table)
startrow <- 0 # Set loop    
overlap <- function(a,b) sum (a == b) / length(b)
df$compare <- NA # overlap values will be inserted here

for (h in 1:length(refs)) {
  refclass <- tmp$prim[tmp$pnum == refs[h]] #subgroup of relevant 'prim'
    x <- length(df$refpat[df$refpat == refs[h]])
    prims <- df$mainprim[startrow:(startrow + x)] # isolate subset from large `df` data.table to reduce memory needed in second loop
      for (i in 1:x) {
      df$compare[startrow + i] <- overlap(prims[i], refclass) 
        }
    startrow <- startrow + x
    print(h)
  }

我使用兩個for循環的原因是為了節省計算機內存。 我可以為每一行使用一個並重新確定refclass ，但這使我的計算機在幾分鍾之內崩潰了。 此循環有效，但大約需要250小時才能完成。 我敢肯定有一些方法可以簡單地從df tmp子集中所需的行，然后對每個origpat重復此操作，但是我的data.table技能不能勝任該任務，而且我找不到答案來解釋如何做到這一點在SO或data.table pdf文件上工作。

任何建議都非常歡迎

EDIt @Frank我要進行的特定比較總是在變化。 主要問題如下。 考慮一個長df具有兩列鏈接的pnum （專利號），一列稱為origpat ，第二列稱為ref.pat 。 每列包含多個重復的pnum但每種組合（在一行上）都是唯一的。 它在公司專利和較早的專利之間建立了聯系。 該數據集大約有22個MIO行。 然后，我有多個其他數據表，例如，一個將pnum鏈接到inventors ，一個將pnum鏈接到各種技術分類。 我感興趣的是找到一種最快的方法，以成對的方式比較鏈接的數據（例如發明人，技術類別），並在df的行中定義該對（即origpat和ref.pat ）。 到目前為止，我擁有的data.table解決方案是最快的，但仍需要數天才能完成一次新的比較。 希望這可以幫助

Answer 1

我想到的最好的主意是：

df[,idx := .I] # Add an index to the data.table to group by row of df
df[,compare := sum(tmp[pnum == ref.pat, prim] == mainprim) /
     length(tmp[pnum == ref.pat,prim]),by = idx]

或重用您的overlap函數（仍使用idx列）：

df[,compare := overlap(
                mainprim,
                tmp[pnum == ref.pat, prim]),
    by=idx]

它的作用是按行分組，然后使用“子集數據”中的列獲取該行的mainprim和所需的tmp子集。

如果要避免創建idx列，則可以使用by=1:nrow(df)代替，但這可能會減慢該過程的速度（在data.table使用實際的列會更快）。

---

@Docendo的重大改進：

您可以通過創建一個中間變量來存儲子集而不是每行重復兩次來進一步加快該過程：

df[,compare := {x = tmp[pnum == ref.pat, prim]; sum(x == mainprim) / length(x)}, by = idx]

如果df存在ref.pat和mainprim重復組合， ref.pat可以通過使用by = list(ref.pat, mainprim)代替by = idx來進一步優化性能：

df[,compare := {x = tmp[pnum == ref.pat, prim]; sum(x == mainprim) / length(x)},
   by = list(ref.pat, mainprim)]

還有另一個可能只是最小的改進可以通過使用mean()而不是sum()/length() ：

df[,compare := mean(tmp[pnum == ref.pat, prim] == mainprim), by = list(ref.pat, mainprim)]

Answer 2

如果我對問題的理解是正確的，那么您只需要在ref.pat上連接這兩個表ref.pat 。 確保班ref.pat在df和pnum在tmp是相同的。 然后，可以通過以下方式獲得所需的聯接：

library(data.table)

df <- data.table(df)
tmp <- data.table(tmp)

setkey(df, 'ref.pat')
out <- df[tmp, nomatch = 0]