Removing duplicates from DataFrame in R

Question

I have this data

UserID   Quiz_answers            Quiz_Date       
  1     `a1,a2,a3`Positive       26-01-2017        
  1     `a1,a4,a3`Positive       26-01-2017        
  1     `a1,a2,a4`Negative       28-02-2017        
  1     `a1,a2,a3`Neutral        30-10-2017        
  1     `a1,a2,a4`Positive       30-11-2017        
  1     `a1,a2,a4`Negative       28-02-2018    

  2     `a1,a2,a3`Negative       27-01-2017            
  2     `a1,a7,a3`Neutral        28-08-2017        
  2     `a1,a2,a5`Negative       28-01-2017  

I want to remove rows that are duplicates:
Rules for rows being duplicates are:

1. The word occuring after backtick(`) in Quiz_answers column are same

2. For such rows if the userID and Quiz_Date column values are also same then the row is duplicate`

     UserID<-c(1,1,1,1,1,1,2,2,2) Quiz_answers<-c("`a1,a2,a3`Positive","`a1,a4,a3`Positive","`a1,a2,a4`Negative","a1,a2,a3`Neutral","`a1,a2,a4`Positive","`a1,a2,a4`Negative","`a1,a2,a3`Negative","`a1,a7,a3`Neutral","`a1,a2,a5`Negative") Quiz_Date<-as.Date(c("26-01-2017","26-01-2017","28-02-2017","30-10-2017","30-11-2017","28-02-2018","27-01-2017","28-08-2017","28-01-2017"),'%d-%m-%Y') data<-data.frame(UserID,Quiz_answers,Quiz_Date) 

-I have written the below code

   data.removeDuplicates<- function(frames)
    {   
         apply(frames[ ,c(grep("UserID", colnames(data)),grep("Quiz_answers", colnames(data)),grep("Quiz_Date", colnames(data)))],1,function(slice){     
             Outcome<-paste0("`",tail(strsplit(slice[2],split="`")[[1]],1))      
             cat("\n\n Searching for records: ",slice[1],Outcome,slice[3])
            data<<-data[!( data$UserID == slice[1] &  paste0("`",sapply(strsplit(as.character(data[,2]),'`'), tail, 1)) == c(Outcome) & data[,3]==c(slice[3])), ]   
        })      
        print(frames)
    }
    data.removeDuplicates(data)
    print(data)
    [1] UserID       Quiz_answers Quiz_Date   
    <0 rows> (or 0-length row.names)

I was expecting output

UserID   Quiz_answers            Quiz_Date       
  1     `a1,a2,a3`Positive       26-01-2017        
  1     `a1,a2,a4`Negative       28-02-2017        
  1     `a1,a2,a3`Neutral        30-10-2017        
  1     `a1,a2,a4`Positive       30-11-2017        
  1     `a1,a2,a4`Negative       28-02-2018    

  2     `a1,a2,a3`Negative       27-01-2017            
  2     `a1,a7,a3`Neutral        28-08-2017        
  2     `a1,a2,a5`Negative       28-01-2017  

Only the second row should get deleted from the DataFrame as per the rule its the only row which satisifies the condition of being duplicate. What am i doing wrong?

Answer 1

Give this a try

Your data

df <- read.table(text="UserID   Quiz_answers            Quiz_Date       
1     `a1,a2,a3`Positive       26-01-2017        
1     `a1,a4,a3`Positive       26-01-2017        
1     `a1,a2,a4`Negative       28-02-2017        
1     `a1,a2,a3`Neutral        30-10-2017        
1     `a1,a2,a4`Positive       30-11-2017        
1     `a1,a2,a4`Negative       28-02-2018    
2     `a1,a2,a3`Negative       27-01-2017            
2     `a1,a7,a3`Neutral        28-08-2017        
2     `a1,a2,a5`Negative       28-01-2017", header = TRUE, stringsAsFactors=FALSE)

Solution & output

library(dplyr)
ans <- df %>%
        mutate(grp = sub(".*`(\\D+)$", "\\1", Quiz_answers)) %>%
        group_by(grp, UserID, Quiz_Date) %>%
        slice(1) %>%
        ungroup() %>%
        select(-grp) %>%
        arrange(UserID, Quiz_Date)

# A tibble: 8 x 3
  # UserID       Quiz_answers  Quiz_Date
   # <int>              <chr>      <chr>
# 1      1 `a1,a2,a3`Positive 26-01-2017
# 2      1 `a1,a2,a4`Negative 28-02-2017
# 3      1 `a1,a2,a4`Negative 28-02-2018
# 4      1  `a1,a2,a3`Neutral 30-10-2017
# 5      1 `a1,a2,a4`Positive 30-11-2017
# 6      2 `a1,a2,a3`Negative 27-01-2017
# 7      2 `a1,a2,a5`Negative 28-01-2017
# 8      2  `a1,a7,a3`Neutral 28-08-2017

Answer 2

You can use sqldf package likes the following. First, find the group of Positive , Negative , and Neutral . Then, filter the duplicate using group by :

require("sqldf")
result <- sqldf("SELECT * FROM df WHERE Quiz_answers LIKE '%`Positive' GROUP BY UserID, Quiz_Date 
       UNION 
       SELECT * FROM df WHERE Quiz_answers LIKE '%`Negative' GROUP BY UserID, Quiz_Date 
       UNION 
       SELECT * FROM df WHERE Quiz_answers LIKE '%`Neutral' GROUP BY UserID, Quiz_Date")

The result after running is:

  UserID       Quiz_answers  Quiz_Date
1      1  `a1,a2,a3`Neutral 30-10-2017
2      1 `a1,a2,a4`Negative 28-02-2017
3      1 `a1,a2,a4`Negative 28-02-2018
4      1 `a1,a2,a4`Positive 30-11-2017
5      1 `a1,a4,a3`Positive 26-01-2017
6      2 `a1,a2,a3`Negative 27-01-2017
7      2 `a1,a2,a5`Negative 28-01-2017
8      2  `a1,a7,a3`Neutral 28-08-2017

Answer 3

Here's a two line solution, using only base R:

data[,"group"] <- with(data, sub(".*`", "", Quiz_answers))

data <- data[as.integer(rownames(unique(data[, !(names(data) %in% "Quiz_answers")   ]))), !(names(data) %in% "group")]