寻找有效的方法来查询R或Stata中的子组观察

Question

I want to write a function that performs an operation on each record in a dataset, based on all other records within a subgroup of unique values [id]. I'm very new to R, but I know that you can query a subset of records based on a condition using the following:

df$date[id == "1234"]

Is it possible to replace "1234" with a variable derived from the unique row that the function is operating on? Something like...

df$date[id == df$id]

, so that it pulls values of [date] where [id] matches [id] of the index row? In practice I would to use this in a loop, where for values of x, I can query a specific [date] value using:

df$date[id == df$id & order == x]

My dataset has multiple records for each unique [id]. Ultimately, I would like to compare the [date_1] value for each record to the [date_2] for all other records that are in each index record's [id] subgroup. The data looks something like this:

[id] | [order] | [date_1] | [date_2] |
-------------------------------------- 
  A  |    1    |    1/1   |    1/30  |
  A  |    2    |    1/5   |    1/5   |
  A  |    3    |    1/7   |    1/8   |
  A  |    4    |    1/9   |    1/9   |
 -------------------------------------
  B  |    1    |    3/7   |    3/10  |
  B  |    2    |    4/1   |    4/9   |
--------------------------------------

Though this could be done by looping through each unique value [id] and then cycling through each unique value [order], the number of records (5-10 million) proves that approach to be extremely slow and resource intensive. I'm wondering if there is a more efficient way to simply loop through the [order] value and then compute this operation for every record simultaneously.

As I said, I'm new to R, so I'm not sure exact syntax of everything yet, but I'm picturing something like this:

for x = 1/max(order){ 
    df$episode_start <- 1 if df$date_1 - df$date_2[id == df$id & order == x] > 1
    }

I can provide more detail on the overall objective of this project, if it would be useful. In short, these data are hospital records, and the goal is to identify records that begin a new segment, which is defined as an encounter that has no prior discharge within 1 day of admission. The data becomes tricky in that there are overlapping records (eg if a patient was an inpatient in long-term care, and had to go for an outpatient visit to the emergency department) -- in the example above A2 and A3 look like they are new encounters based on the discharge date [date_2] of the prior record, however A2, A3 and A4 all occurred during the span of A1, therefore the result should look like this:

[id] | [order] | [date_1] | [date_2] | [episode_start]
------------------------------------------------------ 
  A  |    1    |    1/1   |    1/30  |       1
  A  |    2    |    1/5   |    1/5   |       0
  A  |    3    |    1/7   |    1/8   |       0
  A  |    4    |    1/9   |    1/9   |       0
 -----------------------------------------------------
  B  |    1    |    3/7   |    3/10  |       1
  B  |    2    |    4/1   |    4/9   |       1
------------------------------------------------------

Thanks in advance. Any help or direction is much appreciated. Note: I primarily work in Stata, and attempted to use the -bysort- command to do something similar, but to no avail. Thought maybe R was more suited for this. Open to suggestions using either.

Answer 1

The problem of overlapping hospital stays shows up from time to time on Statalist. See an example here . The solution is to convert the admission/discharge date dyad to long form and to order both events chronologically. A new hospital spell is either the first observation for a patient or if the patient was out of the hospital at the end of the day of the previous observation. Here's an example with data derived from Bulat's R solution (modified to add 2 additional stays):

* Example generated by -dataex-. To install: ssc install dataex
clear
input str1 id byte order str10(date_1 date_2)
"A" 1 "2016-01-01" "2016-01-30"
"A" 2 "2016-01-05" "2016-01-05"
"A" 3 "2016-01-07" "2016-01-08"
"A" 4 "2016-01-09" "2016-01-09"
"A" 5 "2016-02-09" "2016-02-09"
"B" 1 "2016-03-07" "2016-03-10"
"B" 2 "2016-03-08" "2016-03-08"
"B" 3 "2016-04-01" "2016-04-9"
end

gen ndate1 = date(date_1,"YMD")
gen ndate2 = date(date_2,"YMD")
format %td ndate1 ndate2

* confirm that each observation is uniquely identified by
isid id order, sort

* reshape to long; event==1 => admission; event==2 => discharge
reshape long ndate, i(id order) j(event)

* push the discharge date a day later (to make consecutive stays overlap)
replace ndate = ndate + 1 if event == 2

* define an inout increment for admission and discharge events
bysort id order (event): gen inout = cond(_n==1,1,-1)

* for each patient, sort events by date; for multiple events on the same day,
* put admissions before discharge
gsort id ndate -event
by id: gen eventsum = sum(inout)

* if the previous eventsum is 0, a new hospitalization spell starts
by id: gen spell = sum(_n == 1 | eventsum[_n-1] == 0)

* return to the original wide form data
keep if inout == 1

* flag the first obs of each spell
bysort id spell (ndate order): gen newspell = _n == 1

list id order date_1 date_2 spell newspell, sepby(id spell)

and the results:

. list id order date_1 date_2 spell newspell, sepby(id spell)

     +---------------------------------------------------------+
     | id   order       date_1       date_2   spell   newspell |
     |---------------------------------------------------------|
  1. |  A       1   2016-01-01   2016-01-30       1          1 |
  2. |  A       2   2016-01-05   2016-01-05       1          0 |
  3. |  A       3   2016-01-07   2016-01-08       1          0 |
  4. |  A       4   2016-01-09   2016-01-09       1          0 |
     |---------------------------------------------------------|
  5. |  A       5   2016-02-09   2016-02-09       2          1 |
     |---------------------------------------------------------|
  6. |  B       1   2016-03-07   2016-03-10       1          1 |
  7. |  B       2   2016-03-08   2016-03-08       1          0 |
     |---------------------------------------------------------|
  8. |  B       3   2016-04-01    2016-04-9       2          1 |
     +---------------------------------------------------------+

Answer 2

Here is something to get you started using data.table package in R:

data <- read.table(text = "id order date_1 date_2 
A 1 2016-01-01 2016-01-30 
A 2 2016-01-05 2016-01-05
A 3 2016-01-07 2016-01-08
A 4 2016-01-09 2016-01-09
B 1 2016-03-07 2016-03-10
B 2 2016-04-01 2016-04-9", header = T)
library(data.table)
data$date_1 <- as.Date(data$date_1)
data$date_2 <- as.Date(data$date_2)
dt <- data.table(data, key = c("date_1", "date_2"))

res <- foverlaps(dt, dt, by.x = c("date_1", "date_2"), by.y = c("date_1", "date_2"))

# Remove matches from irrelevant groups.
res <- res[id == i.id]

# Find the period start date.
res[, min.date := min(i.date_1), by = .(id, order)]
res[, period.start := (date_1 == min.date)]

# Order records according to the period start date.
res <- res[order(id, order, i.date_1)]
# Remove duplicate rows
res <- res[, .SD[1], by = .(id, order)]

# Print resutls.
res[, .(id, order, date_1, date_2, period.start)][]

#       id order     date_1     date_2 period.start
# 1:  A     1 2016-01-01 2016-01-30         TRUE
# 2:  A     2 2016-01-05 2016-01-05        FALSE
# 3:  A     3 2016-01-07 2016-01-08        FALSE
# 4:  A     4 2016-01-09 2016-01-09        FALSE
# 5:  B     1 2016-03-07 2016-03-10         TRUE
# 6:  B     2 2016-04-01 2016-04-09         TRUE

Answer 3

One convenient way to get to the subsets for processing is to use by . That will automatically subset your data.frame (in this case by ID) and allow you to focus on the handling of the records for each ID.

result <- by(df, df$id, function(x){
              ## identify start dates for sub-group
             })

However, I suspect you'll still find that to be slow. Using data.table as suggested in another answer should help with that.

You could further speed up processing by parallelising this over ID groups.Take a look at the foreach package to help with that. It allows you to write code like this (assuming df$id is a factor):

foreach(i = levels(df$id)) %dopar% {
    ## Identify start dates for group i
}

Answer 4

I'd solve this using the dplyr package, a fantatsic data manipulation tool you can install by running install.packages('dplyr') and then library('dplyr') .

The cheatsheet for this package explains how to manipulate data very eloquently: https://www.rstudio.com/wp-content/uploads/2015/02/data-wrangling-cheatsheet.pdf

I'm not entirely sure what you want to calculate. Are you trying to create a new column with a calculation based on the values in each row? Or, are you trying to calculate something for each unique value of ID ? In the former case, I would use dplyr::mutate(df, newcolumn = some_operation) . In the latter case, I would use group_by(id) and then functions like filter() and summarise() to generate a new dataframe with one row for each ID.