How to join columns with .x, .y suffix in R

Question

I have to create a dataframe containing data from a list of sensors between certain date period:

DATE                SENSOR1 SENSOR2 SENSOR3 SENSOR4
2020-04-20 00:00:00 1015    19.88   95.80   9.020 
2020-04-20 00:10:00 1015    19.84   96.10   8.970 
2020-04-20 00:20:00 1015    19.84   96.40   9.010 
2020-04-20 00:30:00 1015    19.81   96.60   9.210
2020-04-20 00:40:00 1015    19.79   96.80   9.700 
2020-04-20 00:50:00 1015    19.81   97.00   8.870

Initially, I create a dataframe with 1 column ( DATE : rows containing all date between specified dates, with 10 minutes intervals). Normally it can have thousands of rows, but in order to reproduce a example we can keep it simple:

periods <- data.frame(DATE = c("2020-04-20 00:00:00","2020-04-20 00:10:00","2020-04-20 00:20:00","2020-04-20 00:30:00","2020-04-20 00:40:00","2020-04-20 00:50:00"))

I have a list of sensor -> ID, so inside for loop I iterate all the sensors, querying my database returning DATE and VALUE from each one. The problem is, a sensor can have 2 or more id's, depending on which date the data is stored.

ID   SENSORNAME
1     SENSOR1 <- row that has data from SENSOR1 between 2020-04-20 00:00:00 and 2020-04-20 00:20:00
2     SENSOR2 ...
3     SENSOR3 ...
4     SENSOR4 ...
5     SENSOR1 <- row that has data from SENSOR1 between 2020-04-20 00:30:00 and 2020-04-20 00:50:00
6     SENSOR2 ...
7     SENSOR3 ...
8     SENSOR4 ...

Original code:

for (i in 1:length(sensors$ID)) {
  sensor <- dbGetQuery(con, paste0("SELECT DATE, VALUE FROM MEASURES WHERE DATE between '2020-04-20 00:00:00' and '2020-04-20 00:50:00' AND ID= ",sensors$ID[i]," ORDER BY DATE ASC"))
  # getting rid of milliseconds
  sensor$DATE <- as.character(round_date(sensor$DATE, "minute"))
  # Renaming the column with sensor's name
  names(sensor) <- c("DATE", sensors$SENSORNAME[i])

  periods <- merge(periods,sensor,by="DATE",all = TRUE)  

  rm(sensor)
}

Since you can't query my database for data, this example can be reproducible by creating 2 data.frames manually

periods <- data.frame(DATE= c("2020-04-20 00:00:00","2020-04-20 00:10:00","2020-04-20 00:20:00","2020-04-20 00:30:00","2020-04-20 00:40:00","2020-04-20 00:50:00"), SENSOR1= c(1015, 1015, 1015, NA, NA, NA), SENSOR2= c(19.88, 19.84, 19.84, NA, NA, NA), SENSOR3= c(95.80, 96.10, 96.40, NA, NA, NA), SENSOR4= c(9.020, 8.970, 9.010, NA, NA, NA))
sensor <- data.frame(DATE= c("2020-04-20 00:00:00","2020-04-20 00:10:00","2020-04-20 00:20:00","2020-04-20 00:30:00","2020-04-20 00:40:00","2020-04-20 00:50:00"), SENSOR1= c(NA, NA, NA, 1010, 1010, 1010))

After the 4th iteration, it starts to add a suffix on column names, looking something like this:

DATE                SENSOR1.x SENSOR2.x SENSOR3.x SENSOR4.x SENSOR1.y SENSOR2.y SENSOR3.y SENSOR4.y
2020-04-20 00:00:00  1015      19.88     95.80     9.020      NA        NA        NA        NA
2020-04-20 00:10:00  1015      19.84     96.10     8.970      NA        NA        NA        NA 
2020-04-20 00:20:00  1015      19.84     96.40     9.010      NA        NA        NA        NA 
2020-04-20 00:30:00   NA        NA        NA        NA       1015      19.81     96.60     9.210
2020-04-20 00:40:00   NA        NA        NA        NA       1015      19.79     96.80     9.700 
2020-04-20 00:50:00   NA        NA        NA        NA       1015      19.81     97.00     8.870

Any idea on how to merge this properly, or fixing it after the dataframe is generated?

Answer 1

You can use pivot_longer from tidyr to put everything in a column and rbind everything before using pivot_wider to put everything back in wide format. You also need to remove NAs using na.omit() .

library(tidyr)
periods %>%
  pivot_longer(-DATE) %>%
  rbind(sensor %>%
              pivot_longer(-DATE) ) %>%
  na.omit() %>%
  pivot_wider(names_from = name, values_from = value) 

Joining, by = c("DATE", "name", "value")
# A tibble: 6 x 5
  DATE                SENSOR1 SENSOR2 SENSOR3 SENSOR4
  <fct>                 <dbl>   <dbl>   <dbl>   <dbl>
1 2020-04-20 00:00:00    1015    19.9    95.8    9.02
2 2020-04-20 00:10:00    1015    19.8    96.1    8.97
3 2020-04-20 00:20:00    1015    19.8    96.4    9.01
4 2020-04-20 00:30:00    1010    NA      NA     NA   
5 2020-04-20 00:40:00    1010    NA      NA     NA   
6 2020-04-20 00:50:00    1010    NA      NA     NA 

DATA

periods <- data.frame(DATE= c("2020-04-20 00:00:00","2020-04-20 00:10:00","2020-04-20 00:20:00","2020-04-20 00:30:00","2020-04-20 00:40:00","2020-04-20 00:50:00"), SENSOR1= c(1015, 1015, 1015, NA, NA, NA), SENSOR2= c(19.88, 19.84, 19.84, NA, NA, NA), SENSOR3= c(95.80, 96.10, 96.40, NA, NA, NA), SENSOR4= c(9.020, 8.970, 9.010, NA, NA, NA))
sensor <- data.frame(DATE= c("2020-04-20 00:00:00","2020-04-20 00:10:00","2020-04-20 00:20:00","2020-04-20 00:30:00","2020-04-20 00:40:00","2020-04-20 00:50:00"), SENSOR1= c(NA, NA, NA, 1010, 1010, 1010))