Database Link Connection Timed Out

Question

I have a tomcat server calling a stored procedure from a jsp. In the stored procedure I have a query that fills a temp table with data. That temp table is then joined to another table over a dblink to fill another temp table using the hint - DRIVING_SITE . The last temp table is then joined to another table on our database to return a resultset to tomcat.

I'm sorry but I can't really provide a code example for all this, but the issue I'm having is this - After a while of the database link not being used, the first query made using the link will do nothing and return the error:

test.jsp caught exception, closing connection: ORA-02068: following severe error from DATABASE_LINK_NAME
ORA-03135: connection lost contact

Every subsequent query made on the database link within 10 minutes or so of the last call will be fine. The temp table can be huge or small, the amount of data queried seems to make no difference, but the first call after the idle time will get this error probably 75% of the time. Has anyone experienced this issue? If so, are there any resolutions?

The query is structured like so:

INSERT INTO temp_table_2
WITH last_submissions AS (
    SELECT /*+ DRIVING_SITE(some_schema.some_table_1) */
           bs.unique_id,
           CASE WHEN COUNT(bs.unique_id) > 1 THEN 'Y' ELSE 'N' END some_flag,
           MAX(trx.unique_id) last_submission
    FROM   (SELECT unique_id
            FROM   temp_table_1) oids,
           some_schema.some_table_1@DATABASE_LINK bs,
           some_schema.some_table_1@DATABASE_LINK trx
    WHERE  oids.unique_id = bs.unique_id
      AND  bs.non_unique_join_id = trx.non_unique_join_id
    GROUP BY bs.unique_id),
something_relevant AS (
    SELECT /*+ DRIVING_SITE(some_schema.some_table_2) */
           last_value_of_something.unique_id,
           last_value_of_something.some_flag,
           mv.value_description status
    FROM   (
        SELECT /*+ DRIVING_SITE(some_schema.some_table_1) */
               ls.unique_id,
               CASE WHEN COUNT(ls.unique_id) > 1 THEN 'Y' ELSE 'N' END some_flag,
               MAX(prd.prd_some_id) last_submission
        FROM   last_submissions ls,
               some_schema.some_table_1@DATABASE_LINK trx,
               some_schema.some_table_2@DATABASE_LINK prd
        WHERE  ls.last_submission = trx.unique_id
          AND  trx.some_unique_id = prd.some_unique_id (+)
        GROUP BY ls.unique_id) last_value_of_something,
        some_schema.some_table_2@DATABASE_LINK prd,
        some_schema.some_table_3@DATABASE_LINK cs,
        some_schema.some_display_value_table@DATABASE_LINK mv
    WHERE  last_value_of_something.last_submission = prd.prd_some_id (+)
      AND  prd.some_id = cs.some_id (+)
      AND  cs.status_code = mv.value (+)
      AND  mv.value_type (+) = 'SOME_INDICATOR_FOR_DISPLAY_VALUES')
SELECT ls.unique_id unique_id,
       NVL(pr.status, trx.some_code) status,
       CASE WHEN ls.some_flag = 'Y' OR pr.some_flag = 'Y' THEN 'Yes' ELSE 'No' END display_the_flag
FROM   /*+ DRIVING_SITE(some_schema.some_table_1) */
       last_submissions ls,
       some_schema.some_table_1@DATABASE_LINK trx,
       something_relevant pr
WHERE  ls.last_submission = trx.unique_id
  AND  ls.unique_id = pr.unique_id

Answer 1

Do you expect the network between the two database servers to be stable and to allow connections to exist for some time?

When you use a database link, the local server opens up a connection to the remote server. That connection will be kept open as long as your session is open to be used by other queries. If you are seeing connections getting dropped, that often means that there is something in the network (a firewall commonly) that is detecting and killing idle connections. It can also mean that the network between the two servers is simply unstable.

Ideally, you would resolve the problem by fixing whatever underlying network issue you have. If there is a firewall that is killing idle connections, you should be able to modify the firewall configuration to avoid killing these connections for example.

If fixing the infrastructure is not an option, you could close the connection to the remote server after every query (or at least after every query that could be followed by a long idle time)

ALTER SESSION CLOSE DATABASE LINK <<dblink name>>

That means, however, that you would be setting up and tearing down a connection to the remote server potentially on every query-- that is potentially relatively expensive and potentially causes more load on the remote server (depending, of course, on how frequently it happens and how many sessions you might have).

The whole process of pulling data over a database link into a series of temporary tables in order to serve up data to a human using a web application also strikes me as a potentially problematic architecture. Perhaps you have valid reasons for this. But I would be strongly considering using some sort of replication technology (materialized views, Streams, or GoldenGate are the built-in options) rather than pulling data at runtime over database links.