在数据框行之间查找唯一值并将其替换（R）

Question

I have nested data, with ID numbers for within- and cluster level observations. Let's call them L1IDs and L2IDs.

L1ID <- c(1,2,3,4,5,6)
L2ID <- c(11,11,22,22,33,33)

And for both I have a number of variables. We'll call them L1X's and L2X's

L1X1 <- rnorm(6,3,1.1)
L1X2 <- rnorm(6,0,.7)
L2X1 <- c(0,1,1,1,0,0)
L2X2 <- c(Blue,Blue,Red,Red,Green,Red)

Combining the vectors into a dataframe:

df <- data.frame(L1ID,L2ID,L1X1,L1X2,L2X1,L2X2)
df

I have a problem. The values for the 11 and 33 L2ID are not identical. ID 11 has a 1 for the 2nd entry under L2X1 when it should be 0, and ID 33 has Red in the last entry for L2X2 when it should be Green.

L1X values should be different within cluster but not the L2Xs. I need a way to search a large data base by L2ID and find column values that are not identical. Then, replace them with a chosen value. Ideally, this would be a dataframe where each L2ID is a single row and then each column is a logic vector that says True or False if all values in that column, for that L2ID, match. And then replace them all with a same value. So, for ID 11, I need to be able to see that L2X1 does not match for all subjects clustered within it, and that I can replace the 1 with a 0, but that L2X2 all match.

Does that make sense?

My actual dataset (licensed access so I cannot share) is rather large and manually searching this thing for where values do not match is a pain.

So far, my approach has been to eliminate all L1X variables, use dplyr's distinct() function to reduce each row to unique combinations of the L2X variables (each L2ID typically has 2 unique combinations), and then manually searching for discrepancies. Often it's a decimal point in the wrong place.

Update:

To make these sample data more representative of what I am working with, I changed L2X2 to a character vector and added in a 3rd L2ID. Also, I nearly have 200 columns and 9,000 L2IDs (and since most are doubled, it gets to be about 18,000 obs). I'm trying to find a way to not manually specify each column when searching if their values matched. Tried something like the following:

df %>% group_by(L2ID) %>% sapply(identical())

But I have never used the identical() function in Base R so this didn't work. And still working through what to do next. I appreciate the responses so far; I'm going to keep working through this as we go.

Answer 1

I make no promises on performance, but this is one solution, which takes advantage of the rle (run length encoding) function in R. This, of course, assumes that the example data you provided properly implies that the value should be replaced with the most common value in that group.

> L1ID <- c(1,2,3,4,5,6)
> L2ID <- c(11,11,11,22,22,22)
> L1X1 <- rnorm(6,3,1.1)
> L1X2 <- rnorm(6,0,.7)
> L2X1 <- c(0,0,1,1,1,1)
> L2X2 <- c(13,13,13,8,8,9)
> df <- data.frame(L1ID,L2ID,L1X1,L1X2,L2X1,L2X2)
> df
  L1ID L2ID      L1X1         L1X2 L2X1 L2X2
1    1   11 1.9155828  0.287683782    0   13
2    2   11 2.8383669 -0.693942886    0   13
3    3   11 4.7517203  0.419193550    1   13
4    4   22 2.0092141  0.002223136    1    8
5    5   22 1.2546399 -0.457323727    1    8
6    6   22 0.8622906  0.255975868    1    9

> df %>%
     group_by(L2ID) %>%
     mutate(L2X1_r = rle(L2X1)$values[rle(L2X1)$lengths == max(rle(L2X1)$lengths)],
            L2X2_r = rle(L2X2)$values[rle(L2X2)$lengths == max(rle(L2X2)$lengths)]) %>%
     ungroup()
# A tibble: 6 x 8
   L1ID  L2ID      L1X1         L1X2  L2X1  L2X2 L2X1_r L2X2_r
  <dbl> <dbl>     <dbl>        <dbl> <dbl> <dbl>  <dbl>  <dbl>
1     1    11 1.9155828  0.287683782     0    13      0     13
2     2    11 2.8383669 -0.693942886     0    13      0     13
3     3    11 4.7517203  0.419193550     1    13      0     13
4     4    22 2.0092141  0.002223136     1     8      1      8
5     5    22 1.2546399 -0.457323727     1     8      1      8
6     6    22 0.8622906  0.255975868     1     9      1      8

Update

Based on the comments and updated question, I've realized that rle won't work because it assumes the "majority" value has a long run length encoding. This approach fixes this issue, as well as introduces a way to not have to specify every column to be mutated manually.

> L1ID <- c(1,2,3,4,5,6)
> L2ID <- c(11,11,22,22,33,33)
> L1X1 <- rnorm(6,3,1.1)
> L1X2 <- rnorm(6,0,.7)
> L2X1 <- c(0,1,1,1,0,0)
> L2X2 <- c('Blue','Blue','Red','Red','Green','Red')
> df <- data.frame(L1ID,L2ID,L1X1,L1X2,L2X1,L2X2, stringsAsFactors=F)
> df
  L1ID L2ID     L1X1        L1X2 L2X1  L2X2
1    1   11 4.058659  0.12423215    0  Blue
2    2   11 2.922632  0.30954205    1  Blue
3    3   22 2.719407 -0.33382402    1   Red
4    4   22 1.981046 -0.63617811    1   Red
5    5   33 2.570058 -1.39886373    0 Green
6    6   33 4.471551 -0.05489082    0   Red

> replace_with_right_value = function(col) {
+     tbl = table(col)
+     names(tbl)[tbl == max(tbl)]
+ }

> df %>%
     group_by(L2ID) %>%
     mutate_at(vars(matches('L2X')), replace_with_right_value)
     ungroup()
# A tibble: 6 x 6
   L1ID  L2ID     L1X1        L1X2  L2X1  L2X2
  <dbl> <dbl>    <dbl>       <dbl> <chr> <chr>
1     1    11 4.058659  0.12423215     0  Blue
2     2    11 2.922632  0.30954205     1  Blue
3     3    22 2.719407 -0.33382402     1   Red
4     4    22 1.981046 -0.63617811     1   Red
5     5    33 2.570058 -1.39886373     0 Green
6     6    33 4.471551 -0.05489082     0   Red

The replace_with_right_value function takes in a column and returns the most common element in that vector. mutate_at allows you to specify which columns to select, which is done via vars(matches('L2X')) . If the columns do not follow this pattern, you'll need to modify that string a bit. Matches accepts a regular expression, which should prove very helpful in this case. In this case of L2ID , there is not enough information in the question or the data to determine which value to choose for L2X1 when L2ID == 11 or L2X2 when L2ID == 33 . As a result, it returns both. To force it to choose a value, such as the first one, change the function to return names(tbl)[tbl == max(tbl)][1]

Answer 2

Here we check if L2X1 is consistent for L2ID . You can easily add another column using this logic to check L2X2 as well. We simply check if the min and max value of each L2ID is equal, if if those values are not equal, we replace with the min value in L2X1_Fixed .

df %>% group_by(L2ID) %>% mutate(Test= ifelse(min(L2X1)==max(L2X1), TRUE,FALSE)) %>%
      mutate(L2X1_Fixed = ifelse(Test ==FALSE, min(L2X1), L2X1))

# A tibble: 6 x 8
# Groups:   L2ID [2]
   L1ID  L2ID     L1X1        L1X2  L2X1  L2X2  Test L2X1_Fixed
  <dbl> <dbl>    <dbl>       <dbl> <dbl> <dbl> <lgl>      <dbl>
1     1    11 2.355470 -1.53195614     0    13 FALSE          0
2     2    11 3.784859  0.20900278     0    13 FALSE          0
3     3    11 3.339077 -0.19772481     1    13 FALSE          0
4     4    22 2.512764  0.18222493     1     8  TRUE          1
5     5    22 1.176079  0.04175856     1     8  TRUE          1
6     6    22 3.688449 -0.42174624     1     9  TRUE          1