在一个data.frame中查找具有相同数据的列

Question

I have 1 data.frame named A, there are 5000 columns in it. How can I find columns in this data.frame that are equal to each other.

Answer 1

As @John mentioned, there are problems with using duplicated . I would add that transposing the data.frame forces all the data into a same data type before it is even compared with duplicated . On an example, here is a data.frame:

df <- data.frame( a = LETTERS[1:3],
                  b = 1:3,
                  c = as.character(1:3),
                  d = LETTERS[1:3],
                  e = 1:3,
                  f = 1:3)
df
#   a b c d e f
# 1 A 1 1 A 1 1
# 2 B 2 2 B 2 2
# 3 C 3 3 C 3 3

Note that column c is very similar to columns b , e , and f , but not identical because of the different types (character versus numeric). The solution suggested by @Jubbles would disregard these differences.

Instead, it seems more appropriate to use the identical function on the columns of your data.frame. You can compare columns two-by-two using outer :

are.cols.identical <- function(col1, col2) identical(df[,col1], df[,col2])
identical.mat      <- outer(colnames(df), colnames(df),
                            FUN = Vectorize(are.cols.identical))
identical.mat
# [,1]  [,2]  [,3]  [,4]  [,5]  [,6]
# [1,]  TRUE FALSE FALSE  TRUE FALSE FALSE
# [2,] FALSE  TRUE FALSE FALSE  TRUE  TRUE
# [3,] FALSE FALSE  TRUE FALSE FALSE FALSE
# [4,]  TRUE FALSE FALSE  TRUE FALSE FALSE
# [5,] FALSE  TRUE FALSE FALSE  TRUE  TRUE
# [6,] FALSE  TRUE FALSE FALSE  TRUE  TRUE

From here, you can use clustering to identify groups of identical columns (there may be better ways so if you know one, feel free to comment or even edit my answer.)

library(cluster)
distances <- as.dist(!identical.mat)
tree      <- hclust(distances)
cut       <- cutree(tree, h = 0.5)
cut
# [1] 1 2 3 1 2 2

split(colnames(df), cut)
# $`1`
# [1] "a" "d"
# 
# $`2`
# [1] "b" "e" "f"
# 
# $`3`
# [1] "c"

to disregard differences in floating point values, one can use

are.cols.identical <- function(col1,col2) isTRUE(all.equal((df[,col1],df[,col2]))

a more efficient method than clustering for grouping the names of identical columns is

cut <- apply(identical.mat, 1, function(x)match(TRUE, x))
split(colnames(df), cut)

Answer 2

This question is very similar to the one here , with subtle differences yet with the same caveats.

I would again suggest using digest() , as in the following (thanks to @flodel for the data.frame and for a very nice suggestion above)

df <- data.frame( a = LETTERS[1:3],
  b = 1:3,
  c = as.character(1:3),
  d = LETTERS[1:3],
  e = 1:3,
  f = 1:3)

dfDig <- sapply(df, digest)

ansL <- lapply(seq_along(dfDig), function(x) names(which(dfDig == dfDig[x])))

unique(ansL)

# [[1]]
# [1] "a" "d"

# [[2]]
# [1] "b" "e" "f"

# [[3]]
# [1] "c"

This still won't distinguish between 1.0 and 1 , though.

EDIT

As suggested in the comments by @flodel, the following can be used alternatively after creating dfDig

split(colnames(df), vapply(dfDig, match, 1L, dfDig))

Answer 3

How about transposing the dataframe and using duplicated() ?

B <- as.data.frame(t(A))
dup1 <- duplicated(B)
# if you want to identify all duplicated rows
dup2 <- duplicated(B, fromLast = TRUE)
dup_final <- dup1 * dup2
saved_colnames <- colnames(A)[dup_final]