Similarity in R

Question

I have data for patient IDs and hospitals where these patients were treated. I want to calculate Jaccard similarity. Below is the sample data.

HospitalID  CustID
1   1
2   1
1   2
4   2
1   3
2   3
3   3

The calculation of Jaccard Index for (Hospital1,Hospital2) = No. of patients treated by H1 and H2 / Union of patients treated by H1 and H2 . It will be 2/(3+2-2). I need to calculate it for all the combination of hospitals ie (H1,H2) (H1,H3) (H1,H4) (H2,H4) (H3,H4).

In real dataset, I have data for more than 2000 hospitals and 100K insureds. There are many packages available in R which calculates Jaccard distance but I will have to transpose data and put insured IDs in columns which is not feasible as there are more than 100K insureds. Sample R dataset show below -

dt = read.table(header = TRUE, 
text ="HospitalID   CustID
                1   1
                2   1
                1   2
                3   2
                1   3
                2   3
                3   3
                ")

Output should look like below -

Comb1   Comb2   Score
H1  H2  0.67
H1  H3  some_value
H1  H4  some_value
H2  H3  some_value
H2  H4  some_value
H3  H4  some_value

Answer 1

Here is a base R solution that is very direct:

uniHosp <- unique(dt$HospitalID)
myCombs <- combn(uniHosp, 2)

myOut <- data.frame(Comb1 = paste0("H", myCombs[1, ]),
                    Comb2 = paste0("H", myCombs[2, ]),
                    stringsAsFactors = FALSE)

myHosp <- dt$HospitalID
myCust <- dt$CustID

 myOut$Jaccard <- sapply(1:ncol(myCombs), function(x) {
    inA <- myCust[myHosp == myCombs[1, x]]
    inB <- myCust[myHosp == myCombs[2, x]]
    length(intersect(inA, inB))/length(union(inA, inB))
})

 myOut
   Comb1 Comb2   Jaccard
 1    H1    H2 0.6666667
 2    H1    H3 0.6666667
 3    H2    H3 0.3333333

There is probably a much faster approach using data.table or dplyr , but the above should get you started in the right direction.