从SAS中的两个Oracle数据库联接表

Question

I am joining two tables together that are located in two separate oracle databases.

I am currently doing this in sas by creating two libname connections to each database and then simply using something like the below.

libname dbase_a oracle user= etc... ;
libname dbase_b oracle user= etc... ;

proc sql;
create table t1 as 

select a.*, b.*
from dbase_a.table1 a inner join dbase_b.table2 b
on a.id = b.id;
quit;

However the query is painfully slow. Can you suggest any better options to speed up such a query (short of creating a database link going down the path of creating a database link)?

Many thanks for looking at this.

Answer 1

If those two databases are on the same server and you are able to execute cross-database queries in Oracle, you could try using SQL pass-through:

proc sql;
connect to oracle (user= password= <...>);
create table t1 as
select * from connection to oracle (
  select a.*, b.*
  from dbase_a.schema_a.table1 a
  inner join dbase_b.schema_b.table2 b
    on a.id = b.id;
);
disconnect from oracle;
quit;

I think that, in most cases, SAS attemps as much as possible to have the query executed on the database server, even if pass-through was not explicitely specified. However, when that query queries tables that are on different servers, different databases on a system that does not allow cross-database queries or if the query contains SAS-specific functions that SAS is not able to translate in something valid on the DBMS system, then SAS will indeed resort to 'downloading' the complete tables and processing the query locally, which can evidently be painfully inefficient.

Answer 2

The select is for all columns from each table, and the inner join is on the id values only . Because the join criteria evaluation is for data coming from disparate sources, the baggage of all columns could be a big factor in the timing because even non-match rows must be downloaded (by the libname engine, within the SQL execution context) during the ON evaluation.

One approach would be to:

• Select only the id from each table
• Find the intersection
• Upload the intersection to each server (as a scratch table)
• Utilize the intersection on each server as pass through selection criteria within the final join in SAS

There are a couple variations depending on the expected number of id matches, the number of different ids in each table, or knowing table-1 and table-2 as SMALL and BIG. For a large number of id matches that need transfer back to a server you will probably want to use some form of bulk copy. For a relative small number of ids in the intersection you might get away with enumerating them directly in a SQL statement using the construct IN () . The size of a SQL statement could be limited by the database, the SAS/ACCESS to ORACLE engine, the SAS macro system.

Consider a data scenario in which it has been determined the potential number of matching ids would be too large for a construct in (id-1,...id-n) . In such a case the list of matching ids are dealt with in a tabular manner:

libname SOURCE1 ORACLE ....;
libname SOURCE2 ORACLE ....;

libname SCRATCH1 ORACLE ... must specify a scratch schema ...;
libname SCRATCH2 ORACLE ... must specify a scratch schema ...;

proc sql;
    connect using SOURCE1 as PASS1;
    connect using SOURCE2 as PASS2;

    * compute intersection from only id data sent to SAS;
    create table INTERSECTION as
    (select id from connection to PASS1 (select id from table1))
    intersect
    (select id from connection to PASS2 (select id from table2))
    ;

    * upload intersection to each server;
    create table SCRATCH1.ids as select id from INTERSECTION;
    create table SCRATCH2.ids as select id from INTERSECTION;

    * compute inner join from only data that matches intersection;
    create table INNERJOIN as select ONE.*, TWO.* from
    (select * from connection to PASS1 (
        select * from oracle-path-to-schema.table1 
        where id in (select id from oracle-path-to-scratch.ids)
    ))
    JOIN
    (select * from connection to PASS2 (
        select * from oracle-path-to-schema.table2
        where id in (select id from oracle-path-to-scratch.ids)
    ));
    ...

For the case of both table-1 and table-2 having very large numbers of ids that exceed the resource capacity of your SAS platform you will have to also iterate the approach for ranges of id counts. Techniques for range criteria determination for each iteration is a tale for another day.