将一列分成两列并保留分隔符

Question

I have a very large data array:

'data.frame':   40525992 obs. of  14 variables:    
 $ INSTNM     : Factor w/ 7050 levels "A   W Healthcare Educators"     
 $ Total      : Factor w/ 3212 levels "1","10","100",    
 $ Crime_Type : Factor w/ 72 levels "MURD11","NEG_M11",    
 $ Count      : num  0 0 0 0 0 0 0 0 0 0 ...

The Crime_Type column contains the type of Crime and the Year, so "MURD11" is Murder in 2011. These are college campus crime statistics my kid is analyzing for her school project, I am helping when she is stuck. I am currently stuck at creating a clean data file she can analyze

Once i converted the wide file (all crime types '9' in columns) to a long file using 'gather' the file size is going from 300MB to 8 GB. The file I am working on is 8GB. do you that is the problem. How do i convert it to a data.table for faster processing?

What I want to do is to split this 'Crime_Type' column into two columns 'Crime_Type' and 'Year'. The data contains alphanumeric and numbers. There are also some special characters like NEG_M which is 'Negligent Manslaughter'.

We will replace the full names later but can some one suggest on how I separate

MURD11 --> MURD and 11 (in two columns) NEG_M10 --> NEG_M and 10 (in two columns)

etc...

I have tried using,

df <- separate(totallong, Crime_Type, into = c("Crime", "Year"), sep = "[:digit:]", extra = "merge")
df <- separate(totallong, Crime_Type, into = c("Year", "Temp"), sep = "[:alpha:]", extra = "merge")

The first one separates the Crime as it looks for numbers. The second one does not work at all.

I also tried

df$Crime_Type<- apply (strsplit(as.character(df$Crime_Type), split="[:digit:]"))

That does not work at all. I have gone through many posts on stack-overflow and thats where I got these commands but I am now truly stuck and would appreciate your help.

Answer 1

Since you're using tidyr already (as evidenced by separate ), try the extract function, which, given a regex, puts each captured group into a new column. The 'Crime_Type' is all the non-numeric stuff, and the 'Year' is the numeric stuff. Adjust the regex accordingly.

library(tidyr)
extract(df, 'Crime_Type', into=c('Crime', 'Year'), regex='^([^0-9]+)([0-9]+)$')

Answer 2

In base R , one option would be to create a unique delimiter between the non-numeric and numeric part. We can capture as a group the non-numeric ( [^0-9]+ ) and numeric ( [0-9]+ ) characters by wrapping it inside the parentheses ( (..) ) and in the replacement we use \\\\1 for the first capture group, followed by a , and the second group ( \\\\2 ). This can be used as input vector to read.table with sep=',' to read as two columns.

 df1 <- read.table(text=gsub('([^0-9]+)([0-9]+)', '\\1,\\2', 
                   totallong$Crime_Type),sep=",", col.names=c('Crime', 'Year'))
 df1
 #   Crime Year
 #1  MURD   11
 #2 NEG_M   11

If we need, we can cbind with the original dataset

cbind(totallong, df1)

Or in base R , we can use strsplit with split specifying the boundary between non-number ( (?<=[^0-9]) ) and a number ( (?=[0-9]) ). Here we use lookarounds to match the boundary. The output will be a list , we can rbind the list elements with do.call(rbind and convert it to data.frame

as.data.frame(do.call(rbind, strsplit(as.character(totallong$Crime_Type), 
                        split="(?<=[^0-9])(?=[0-9])", perl=TRUE)))
#     V1 V2
#1  MURD 11
#2 NEG_M 11

---

Or another option is tstrsplit from the devel version of data.table ie. v1.9.5 . Here also, we use the same regex . In addition, there is option to convert the output columns into different class .

library(data.table)#v1.9.5+
setDT(totallong)[, c('Crime', 'Year') := tstrsplit(Crime_Type, 
    "(?<=[^0-9])(?=[0-9])",  perl=TRUE, type.convert=TRUE)]
#   Crime_Type Crime Year
#1:     MURD11  MURD   11
#2:    NEG_M11 NEG_M   11

If we don't need the 'Crime_Type' column in the output, it can be assigned to NULL

totallong[, Crime_Type:= NULL]

NOTE: Instructions to install the devel version are here

---

Or a faster option would be stri_extract_all from library(stringi) after collapsing the rows to a single string ('v2'). The alternate elements in 'v3' can be extracted by indexing with seq to create new data.frame

library(stringi)
v2 <- paste(totallong$Crime_Type, collapse='')
v3 <- stri_extract_all(v2, regex='\\d+|\\D+')[[1]]
ind1 <- seq(1, length(v3), by=2)
ind2 <- seq(2, length(v3), by=2)
d1 <- data.frame(Crime=v3[ind1], Year= v3[ind2])

Benchmarks

v1 <- do.call(paste, c(expand.grid(c('MURD', 'NEG_M'), 11:15), sep=''))
set.seed(24)
test <- data.frame(v1= sample(v1, 40525992, replace=TRUE ))

system.time({
   v2 <- paste(test$v1, collapse='')
   v3 <- stri_extract_all(v2, regex='\\d+|\\D+')[[1]]
   ind1 <- seq(1, length(v3), by=2)
   ind2 <- seq(2, length(v3), by=2)
   d1 <- data.frame(Crime=v3[ind1], Year= v3[ind2])
 })
 #user  system elapsed 
 #56.019   1.709  57.838 

data

totallong <- data.frame(Crime_Type= c('MURD11', 'NEG_M11'))