How to use apply function to calculate the distance between two matrices

Question

I'm trying to calculate the euclidean distance between two matrices. I have already achieved that using 2 for loops but trying to vectorize the calculation to speed up. I'm using pdist as a benchmark to valid if the distance is calculated correctly.

Thanks to this post, https://medium.com/@souravdey/l2-distance-matrix-vectorization-trick-26aa3247ac6c , I tried to achieve the same thing in r with this code:

dist <- sqrt(rowSums(xtest**2)+rowSums(xtrain**2)-2*xtrain %*% t(xtest))

But the result is different from what comes out of pdist. I am not sure what's wrong with this.

Here are some codes

Create some data

xtest=matrix(cbind(c(0,0),c(1,31)),2,2,byrow=TRUE)
xtrain=matrix(cbind(c(9,2),c(4,15),c(7,8),c(-22,-2)),4,2,byrow=TRUE)

Calculate using double loops

mydist <- function(xtest,xtrain) {
  euc.dist <- function(x1, x2) sqrt(sum((x1 - x2) ^ 2))
  dist <- matrix(,nrow=nrow(xtrain),ncol=nrow(xtest))
  for (i in 1:nrow(xtrain)){
    for (j in 1:nrow(xtest)){
      dist[i,j] <- euc.dist(xtrain[i,], xtest[j,])
    }
  }
  return (dist)
}
> mydist(xtest,xtrain)
          [,1]     [,2]
[1,]  9.219544 30.08322
[2,] 15.524175 16.27882
[3,] 10.630146 23.76973
[4,] 22.090722 40.22437

The result is same as using pdist

> libdists <- pdist(xtrain,xtest)
> as.matrix(libdists)
          [,1]     [,2]
[1,]  9.219544 30.08322
[2,] 15.524175 16.27882
[3,] 10.630146 23.76973
[4,] 22.090721 40.22437

But if I use matrix multiplication method it's wrong

> mydist2 <- function(xtest,xtrain) {
+   dist <- sqrt(rowSums(xtest**2)+rowSums(xtrain**2)-2*xtrain %*% t(xtest))
+   return (dist)
+ }
> mydist2(xtest,xtrain)
          [,1]     [,2]
[1,]  9.219544      NaN
[2,] 34.684290 16.27882
[3,] 10.630146      NaN
[4,] 38.078866 40.22437

I have also tried to use mapply function

> mydist3 <- function(xtest,xtrain) {
+   euc.dist <- function(x1, x2) sqrt(sum((x1 - x2) ^ 2))
+   dist <- mapply(euc.dist, xtest,xtrain)
+   return (dist)
+ }
> mydist3(xtest,xtrain)
[1]  9  3  7 53  2 14  8 33

I think it goes element wise rather than takes each row as a vector to calculate the distance between two vectors.

Any suggestions will be appreciated!

Answer 1

Use two apply instances with the second nested in the first:

d1 <- apply(xtest, 1, function(x) apply(xtrain, 1, function(y) sqrt(crossprod(x-y))))

Check against pdist :

library(pdist)
d2 <- as.matrix(pdist(xtrain, xtest))

all.equal(d1, d2, tolerance = 1e-7)
## [1] TRUE