将R中的时间序列数据从秒转换为小时均值

Question

Note:I have re-framed the previous question as told in the comments.

I am using three different packages,ie, dplyr, data.table and xts to aggregate my seconds data to hourly mean representation. But, to my surprise xts behaves differently as compared to other two. Issues with xts are:

• Results in one extra observation as compared to other two
• Hourly mean calculated is totally different than the other two

Here is the condensed code for your testing purposes:

library(xts)
library(data.table)
library(dplyr)
t2 <- as.POSIXct(seq(from = 1438367408, to = 1440959383, by = 30), origin = "1970-01-01")
dframe <- data.frame(timestamp=t2, power=rnorm(length(t2)))
#using xts
x <- xts(dframe$power,dframe$timestamp)
h1 <- period.apply(x, endpoints(x, "hours"), mean)
h1 <- data.frame(timestamp=trunc(index(h1),'hours'), power=coredata(h1))
#using data.table
h2 <- setDT(dframe)[, list(power= mean(power)) ,(timestamp= as.POSIXct(cut(timestamp, 'hours')))]
#using dpylr
h3 <- dframe %>% group_by(timestamp= as.POSIXct(cut(timestamp, 'hour'))) %>% summarise(power=mean(power))

Outputs in regard to size:

> dim(h1)
[1] 721   2
> dim(h2)
[1] 720   2
> dim(h3)
[1] 720   2

Outputs in regard to Hourly means:

> head(h1)
            timestamp       power
1 2015-08-01 00:00:00  0.04485894
2 2015-08-01 01:00:00 -0.02299071
> head(h2) # equals to head(h2)
             timestamp       power
1: 2015-08-01 00:00:00  0.10057538
2: 2015-08-01 01:00:00 -0.07456292

Extra observation in case of h1:

> tail(h1)
              timestamp        power
719 2015-08-30 22:00:00  0.069544538
720 2015-08-30 23:00:00  0.011673835
721 2015-08-30 23:00:00 -0.053858563

Clearly for the last hour of day there are two observation. Normally, there should be only one.

My system information:

> sessionInfo()
R version 3.2.2 (2015-08-14)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: OS X 10.10.3 (Yosemite)

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] dplyr_0.4.3      data.table_1.9.7 xts_0.9-7        zoo_1.7-12      

loaded via a namespace (and not attached):
 [1] lazyeval_0.1.10 magrittr_1.5    R6_2.1.1        assertthat_0.1  parallel_3.2.2  DBI_0.3.1       tools_3.2.2    
 [8] Rcpp_0.12.1     grid_3.2.2      chron_2.3-47    lattice_0.20-33

Note:

• Original dataset can be found at the link
• I want a solution to this issue, because in my implementation scenario xts is nearly 35 times faster than the remaining two

Answer 1

This looks like it might be a bug in endpoints because your local timezone is not a full hour offset from UTC. I can replicate the issue if I set my local timezone to yours.

R> Sys.setenv(TZ="Asia/Kolkata")
R> x <- xts(dframe$power,dframe$timestamp)
R> h <- period.apply(x, endpoints(x, "hours"), mean)
R> head(h)
                        [,1]
2015-08-01 00:29:31 124.9055
2015-08-01 01:29:31 129.7197
2015-08-01 02:29:31 139.0899
2015-08-01 03:29:32 145.6592
2015-08-01 04:29:32 153.6840
2015-08-01 05:29:32 114.4809

Note that the endpoints are at half-hour increments, rather than at the end of the hour. This is because Asia/Kolkata is UTC+0530 and endpoints does all its calculations on times represented in UTC.

You can avoid this by explicitly setting the timezone for the POSIXct object to UTC.

require(xts)
require(dplyr)
require(data.table)
Sys.setenv(TZ="Asia/Kolkata")

dframe <- read.csv("~/ap601.csv",head=TRUE,sep=",")
# set timezone on POSIXct object
dframe$timestamp <- as.POSIXct(dframe$timestamp, tz="UTC")

#using xts
x <- xts(dframe$power, dframe$timestamp)
h <- period.apply(x, endpoints(x, "hours"), mean)
h1 <- data.frame(timestamp=trunc(index(h),'hours'), power=coredata(h))
# using data.table
h2 <- setDT(dframe)[, list(power= mean(power)) ,(timestamp= cut(timestamp, 'hour'))]
# using dplyr
h3 <- dframe %>% group_by(timestamp= cut(timestamp, 'hour')) %>% summarise(power=mean(power))

all.equal(h1$power, h2$power)  # TRUE
all.equal(h1$power, h3$power)  # TRUE

---

Here's a work-around to get the same results without setting the timezone for the POSIXct column to UTC. Note that this may not work for timezones with Daylight Saving Time (Asia/Kolkata does not observe any DST).

Basically, the idea is to subtract half an hour from the local time when calculating the endpoints , so that the underlying UTC time aligns on the hour.

dframe <- read.csv("~/ap601.csv",head=TRUE,sep=",")
dframe$timestamp <- as.POSIXct(dframe$timestamp)

# subtract half an hour from the index when calculating endpoints
h <- period.apply(x, endpoints(index(x)-3600*0.5, 'hours'), mean)
h1 <- data.frame(timestamp=trunc(index(h),'hours'), power=coredata(h))
all.equal(h1$power, h2$power)  # TRUE
all.equal(h1$power, h3$power)  # TRUE