优化SAS Proc SQL查询

Question

I have 2 large tables that I am trying to join in order to group the first records based on a field from the second table. The left table has approx.50 mil records of events, the right table has approx.35 mil records of monthly intervals. The monthly intervals are at subjID level, thus I cannot reduce the size of the right table by keeping only start and end dates. Currently it takes about 40 - 60 minutes to perform the join.

I tried to create simple indexes on subjID, eventDate, startDate and endDate, however it did not seem to improve performance (creating the indexes completed in about 5 minutes, join completed in 38 minutes).

Is there any other option I could use to improve processing?

Left Table of events at subjID level:

data eventsTable;
input @1 subjID 8.
    @10 eventDate date9.;
format eventDate mmddyy10.;
datalines;
101      01AUG2011
101      28AUG2011
101      30AUG2011
101      01SEP2011
101      12SEP2011
101      28SEP2011
102      01JAN2015
102      15JAN2015
102      01FEB2015
102      16FEB2015
;
run;

Right Table of monthly intervals at subjID level. I am trying to bring endDate to events if the events occurred between start and end date:

data monthlyTable;
input @1 subjID 8.
    @10 startDate date9. 
    @22 endDate date9.;
format startDate endDate mmddyy10.;
datalines;
101      28JUL2011   30AUG2011
101      30AUG2011   28SEP2011
101      28SEP2011   28OCT2011
102      01DEC2014   02JAN2015
102      02JAN2015   02FEB2015
102      02FEB2015   02MAR2015
;
run;

Output:

proc sql;
create table wantTable as 
    select a.*,
        endDate as monthlyDate
    from eventsTable a left join monthlyTable b on 
        a.subjID = b.subjID
    where a.eventDate > b.startDate and a.eventDate <= b.endDate
        order by subjID, eventDate;
quit;

Answer 1

If you have enough memory and you only need the enddate from monthlyTable , you might find that a format merge is a more efficient way of doing this. However, if both datasets are large, there's only so much optimisation you can hope for as you always have to do at least full read of each.

data t_format(keep = fmtname--hlo) /view = t_format;
  set monthlytable(keep = subjID startdate enddate) end = eof;
  retain fmtname 'myinfmt' type 'i';
  length start end $18; /*Increase for IDs longer than 8 digits*/
  start = cats(put(subjID,z8.),put(startdate + 1,yymmdd10.));
  end   = cats(put(subjID,z8.),put(enddate,yymmdd10.));
  label = enddate;
  output;
  if eof then do;
    hlo = 'O';
    label = .N;
    output;
  end;
run;

proc format cntlin = t_format;
run;

data want;
  set eventstable;
  enddate = input(cats(put(subjID,z8.),put(eventdate,yymmdd10.)),myinfmt18.);
  format enddate yymmdd10.;
run;

Note the use of the yymmdd10. and z8. formats - these ensure that keys are always the same length, avoiding ambiguity, and that the ranges of lookup values are correctly specified in ascending order when creating the numeric informat myinfmt . I suppose, strictly speaking, this is an informat merge rather than a format merge, but it's the same sort of idea.

If you want to return multiple lookup variables via this approach, you'll need to concatenate them together when defining the format and then split them after applying it.

I would estimate that this approach requires about 1.5GB of memory for the datasets you've specified - ie (18 bytes x 2 per date range + 8 bytes for the formatted value) x 35m rows. Depending on the length of your IDs this may differ a bit.

If you need multiple lookup values then you can do a similar thing using a hash merge, but I suspect the format merge is more efficient in this case.

One possible hash merge approach looks like this:

data t_lookup /view= t_lookup;
  set monthlytable;
  by subjID;
  if first.subjID then id_range_count = 0;
  id_range_count + 1;
run;

data want;
  set eventstable;
  if _n_ = 1 then do;
    if 0 then set monthlytable(keep = subjID startdate enddate); /*Add extra lookup vars here as needed*/
    declare hash h(dataset:"t_lookup");
    rc = h.definekey("subjID","id_range_count");
    rc = h.definedata("startdate","enddate"); /*Add extra lookup vars here as needed*/
    rc = h.definedone();
  end;
  match = 0;
  rc    = 0;
  do id_range_count = 1 by 1 while(rc = 0 and match = 0);
    rc = h.find();
    match = startdate < eventdate <= enddate;
  end;
  if match = 0 then call missing(startdate,enddate);
  drop rc match id_range_count;
run;

Answer 2

The best index for your query is a composite index on monthlyTable(subjId, startDate, endDate) . I'm not sure if it will be a big improvement in terms of performance in SAS, however.

Answer 3

I've had better luck with pre-sorting data sets than with creating indexes. However, pre-sorting can take a long time depending on the size of the data sets and what you are sorting them on. It can take longer than the original SQL query, so testing becomes important.

Try running

PROC SORT DATA=eventsTable ;
  BY subjID eventDate ;
RUN ;

PROC SORT DATA=monthlyTable ;
  BY subjID startDate endDate ;
RUN ;

before your PROC SQL. The only explanation I have is that SAS recognizes the SORT BY header information and doesn't need to scan entire tables looking for joins since a given subjID would probably only be on a few consecutive pages. Being on a few consecutive pages also decreases I/O.