如何通过两个属性过滤大型数据集并拆分为子集？ R/Grep

Question

I found myself at the limits of the grep() function or perhaps there are efficient ways of doing this.

Start off a sample data-frame:

Date <- c( "31-DEC-2014","31-DEC-2014","31-DEC-2014","30-DEC-2014",
           "30-DEC-2014","30-DEC-2014", "29-DEC-2014","29-DEC-2014","29-DEC-2014" )

ISIN <- c("LU0168343191", "TW0002418001", "GB00B3FFY088","LU0168343191",
          "TW0002418001", "GB00B3FFY088","LU0168343191", "TW0002418001", "GB00B3FFY088")


price <-c(seq(1:9))

df <- as.data.frame(cbind(Date, ISIN, price))

And the desired Result is a list() containing subsets of the main data file which looks like the below (x3 for the 3 individual Identifiers in Result_I )

The idea is that the data should first filter by ISIN and then filter by Date. this 2 step process should keep my data intact.

Result_d <- c("31-DEC-2014", "30-DEC-2014","29-DEC-2014")
Result_I <- c("LU0168343191","LU0168343191","LU0168343191")
Result_P <- c(1,4,7)

Result_df <- cbind(Result_d, Result_I, Result_P)

Please keep in mid the above is for demo purposes and the real data-set has 5M rows and 50 columns over a period of 450+ different dates as per Result_d so i am lookign for something that is applicable irrespective of nrow or ncol

What i have so far:

I take all unique dates and store:

Unique_Dates <- unique(df$Date)

The same for the Identifiers:

Unique_ID <- unique(df$ISIN)

Now the grepping issue:

If i wanted all rows containing Unique_Dates i would do something like:

pattern <- paste(Unique_dates, collapse = "|")

result <- as.matrix(df[grep(pattern, df$Date),])

and this will retrieve basically the entire data set. i am wondering if anyone knows an efficient way of doing this.

Thanks in advance.

Answer 1

Using dplyr :

library(dplyr)

Date <- c( "31-Dec-2014","31-Dec-2014","31-Dec-2014","30-Dec-2014",
           "30-Dec-2014","30-Dec-2014", "29-Dec-2014","29-Dec-2014","29-Dec-2014" )

ISIN <- c("LU0168343191", "TW0002418001", "GB00B3FFY088","LU0168343191",
          "TW0002418001", "GB00B3FFY088","LU0168343191", "TW0002418001", "GB00B3FFY088")


price <-c(seq(1:9))

DF <- data.frame(Date, ISIN, price,stringsAsFactors=FALSE)
DF$Date=as.Date(DF$Date,"%d-%b-%Y")



#Examine data ranges and frequencies

#date range
range(DF$Date)

#date frequency count
table(DF$Date)

#ISIN frequency count
table(DF$ISIN)


#select ISINs for filtering, with user defined choice of filters

# numISIN = 2
# subISIN = head(names(sort(table(DF$ISIN))),numISIN)


subISIN = names(sort(table(DF$ISIN)))[2]


subDF=DF %>%
dplyr::group_by(ISIN) %>%
dplyr::arrange(ISIN,Date) %>%
dplyr::filter(ISIN %in% subISIN) %>%  
as.data.frame()

#> subDF
#        Date         ISIN price
#1 2014-12-29 LU0168343191     7
#2 2014-12-30 LU0168343191     4
#3 2014-12-31 LU0168343191     1

Answer 2

We convert the 'data.frame' to 'data.table' ( setDT(df) ), grouped by 'Date', specify the 'i' based on the index returned with grep and Subset the Data.table ( .SD ) based on the 'i' index.

library(data.table)
setDT(df)[grep("LU", ISIN), .SD, by = Date]
#           Date         ISIN price
#1: 31-DEC-2014 LU0168343191     1
#2: 30-DEC-2014 LU0168343191     4
#3: 29-DEC-2014 LU0168343191     7