Reshaping data to use for panel data studies, using tidyr or dplyr

Question

I have a data frame, with 28 rows and 26 columns. Each column represents a day, and each row represents a time period t . It looks like this:

dput(head(training_data,10))
structure(list(X11950.0 = c(1L, 15L, 15L, 18L, 31L, 57L, 96L, 
134L, 235L, 291L), X11952.0 = c(0L, 15L, 11L, 18L, 45L, 76L, 
97L, 115L, 200L, 234L), X11955.0 = c(0L, 13L, 14L, 10L, 30L, 
49L, 86L, 114L, 193L, 239L), X11956.0 = c(0L, 15L, 10L, 8L, 38L, 
41L, 96L, 117L, 222L, 227L), X11957.0 = c(0L, 10L, 8L, 14L, 33L, 
61L, 84L, 143L, 211L, 249L), X11958.0 = c(0L, 18L, 14L, 16L, 
48L, 49L, 87L, 117L, 209L, 258L), X11959.0 = c(0L, 14L, 14L, 
17L, 33L, 57L, 93L, 100L, 189L, 241L), X11962.0 = c(0L, 8L, 8L, 
16L, 29L, 48L, 88L, 126L, 202L, 211L), X11963.0 = c(0L, 8L, 10L, 
14L, 44L, 51L, 98L, 148L, 228L, 218L), X11964.0 = c(0L, 15L, 
16L, 12L, 39L, 59L, 109L, 138L, 227L, 233L), X11965.0 = c(0L, 
14L, 13L, 18L, 34L, 46L, 89L, 154L, 199L, 264L), X11966.0 = c(0L, 
18L, 13L, 15L, 37L, 49L, 93L, 142L, 183L, 236L), X11969.0 = c(0L, 
7L, 10L, 15L, 34L, 34L, 84L, 136L, 206L, 233L), X11970.0 = c(0L, 
12L, 14L, 25L, 36L, 50L, 121L, 145L, 206L, 238L), X11971.0 = c(0L, 
17L, 8L, 20L, 36L, 47L, 92L, 108L, 144L, 147L), X12088.0 = c(0L, 
5L, 10L, 9L, 35L, 55L, 92L, 112L, 215L, 241L), X12089.0 = c(1L, 
8L, 10L, 15L, 15L, 0L, 46L, 118L, 221L, 241L), X12090.0 = c(0L, 
8L, 10L, 15L, 46L, 43L, 101L, 106L, 225L, 235L), X12091.0 = c(0L, 
5L, 13L, 19L, 40L, 40L, 80L, 119L, 214L, 208L), X12092.0 = c(0L, 
9L, 13L, 18L, 41L, 41L, 89L, 100L, 205L, 200L), X12095.0 = c(0L, 
8L, 13L, 6L, 32L, 55L, 69L, 112L, 199L, 202L), X12096.0 = c(0L, 
7L, 12L, 27L, 40L, 36L, 102L, 119L, 216L, 228L), X12097.0 = c(0L, 
9L, 15L, 14L, 43L, 51L, 94L, 122L, 220L, 231L), X12098.0 = c(0L, 
10L, 15L, 14L, 47L, 39L, 90L, 117L, 223L, 221L), X12099.0 = c(0L, 
9L, 13L, 16L, 39L, 52L, 94L, 110L, 220L, 216L), X12102.0 = c(0L, 
9L, 10L, 9L, 51L, 43L, 82L, 98L, 175L, 196L)), .Names = c("X11950.0", 
"X11952.0", "X11955.0", "X11956.0", "X11957.0", "X11958.0", "X11959.0", 
"X11962.0", "X11963.0", "X11964.0", "X11965.0", "X11966.0", "X11969.0", 
"X11970.0", "X11971.0", "X12088.0", "X12089.0", "X12090.0", "X12091.0", 
"X12092.0", "X12095.0", "X12096.0", "X12097.0", "X12098.0", "X12099.0", 
"X12102.0"), row.names = c(NA, 10L), class = "data.frame")

I'm trying to do regression analysis, regressing each time period in a given day on its previous 3 values. For that, I want to mung the data in a way that for each row, column 1 yt would contain the value for the time t and columns 2 to 4 would contain y(t-1) till y(t-3) .

My code so far is this:

a <- stack(training_data)[,1, drop=FALSE]
panel.data <- read.csv(text="indiv, t, yt, y.t.1, y.t.2, y.t.3", 
                   colClasses = c(integer(), integer(), integer(),
                                  integer(), integer(), integer()),
                   stringsAsFactors = FALSE )
for(block.id in (0: ((nrow(a)/28) -1))) {
  # day blocks
  block <- a[(28*(block.id)+1) : (28*(block.id+1)), ]
  yt <- block[4:28]
  first.time.period <- block[3:27]
  second.time.period <- block[2:26]
  third.time.period <- block[1:25]

  insert.block <- c((25*(block.id)+1) : (25*(block.id+1))) 
  panel.data[insert.block, ]$yt <- yt
  panel.data[insert.block, ]$y.t.1 <- first.time.period
  panel.data[insert.block, ]$y.t.2 <- second.time.period
  panel.data[insert.block, ]$y.t.3 <- third.time.period

}

Remember again that there are 28 time periods for each day.

This works, but it was a lot of struggle to make it, and it's ugly and not really reusable. For example, now I have another data set which looks like this but has different number of rows, and I want to join this table with another table, etc. It's a headache.

My question: Is there a way to do this, preferably with Hadley's packages (tidyr, dplyr) more cleanly? I've studied these two a bit, and maybe spread() would do it? I'm a beginner in those packages and even though I've tried for a few hours now I haven't been able to solve this.

Answer 1

Here is a solution. I am assuming that you don't want to overlap days (ie. that t-1 for the first period of the second day should not be the last period of the first day). If this is not the case, just remove the group_by() statement. The slice() removes the first three periods of each day.

gather(df,day,y) %>% 
  group_by(day) %>% 
  mutate(t1=lag(y),
         t2=lag(t1),
         t3=lag(t2)) %>% 
  slice(-c(1:3))