Teradata EXPLAIN 中的 STAT FUNCTION 是什么意思？

Question

When I use TOP in a select to get sample data from a Teradata table, it uses a LOT more spool (and hence often I spool out) than using SAMPLE for instance.

Looking at the EXPLAINs to see what the difference in processing is between SAMPLE vs TOP, it seems there is lot more copying of spool tables going for TOP; but the bit I'm confused about is where it says it does a "STAT FUNCTION" step. Can anyone explain what this step is? Below are the two explains. The Primary Index of the table is UNIQUE PRIMARY INDEX (Customer_ID). The Teradata version is 16.10.05.03.

Explain     
SELECT      TOP 2000 
            M.Customer_ID
            , M.customer_type

from        ESRE.MEAS_CUST_TBL as M
WHERE       M.Customer_ID is not null;

  1) First, we lock ESRE.M for read on a reserved RowHash to
     prevent global deadlock.
  2) Next, we lock ESRE.M for read.
  3) We do an all-AMPs RETRIEVE step from ESRE.M by way of an
     all-rows scan with a condition of ("NOT (ESRE.M.Customer_ID IS
     NULL)") into Spool 2 (all_amps), which is built locally on the
     AMPs.  The size of Spool 2 is estimated with high confidence to be
     43,384,684 rows (1,778,772,044 bytes).  The estimated time for
     this step is 1.28 seconds.
  4) We do an all-AMPs STAT FUNCTION step from Spool 2 by way of an
     all-rows scan into Spool 5, which is built locally on the AMPs.
     The result rows are put into Spool 1 (group_amps), which is built
     locally on the AMPs.  This step is used to retrieve the TOP 2000
     rows.  One AMP is randomly selected to retrieve 2000 rows.
     If this step retrieves less than 2000 rows, then execute step 5.
     The size is estimated with high confidence to be 2,000 rows (
     94,000 bytes).
  5) We do an all-AMPs STAT FUNCTION step from Spool 2 (Last Use) by
     way of an all-rows scan into Spool 5 (Last Use), which is
     redistributed by hash code to all AMPs.  The result rows are put
     into Spool 1 (group_amps), which is built locally on the AMPs.
     This step is used to retrieve the TOP 2000 rows.  The size is
     estimated with high confidence to be 2,000 rows (94,000 bytes).
  6) Finally, we send out an END TRANSACTION step to all AMPs involved
     in processing the request.
  -> The contents of Spool 1 are sent back to the user as the result of
     statement 1.



Explain     
SELECT      M.Customer_ID
            , M.customer_type

from        ESRE.MEAS_CUST_TBL as M

WHERE       M.Customer_ID is not null

sample      2000;

  1) First, we lock ESRE.M for read on a reserved RowHash to
     prevent global deadlock.
  2) Next, we lock ESRE.M for read.
  3) We do an all-AMPs RETRIEVE step from ESRE.M by way of an
     all-rows scan with a condition of ("NOT (ESRE.M.Customer_ID IS
     NULL)") into Spool 2 (all_amps), which is built locally on the
     AMPs.  The size of Spool 2 is estimated with high confidence to be
     43,384,684 rows (2,039,080,148 bytes).  The estimated time for
     this step is 1.28 seconds.
  4) We do an all-AMPs SAMPLING step from Spool 2 (Last Use) by way of
     an all-rows scan into Spool 1 (group_amps), which is built locally
     on the AMPs.  Samples are specified as a number of rows.  The size
     of Spool 1 is estimated with high confidence to be 2,000 rows (
     94,000 bytes).
  5) Finally, we send out an END TRANSACTION step to all AMPs involved
     in processing the request.
  -> The contents of Spool 1 are sent back to the user as the result of
     statement 1.

Answer 1

There's a common characteristic of both SAMPLE and TOP, they are executed as the last step in Explain. Thus your WHERE-condition is applied first, spooling almost all rows.

A simple workaround to avoid spooling large amounts of rows is nested sampling (similar for TOP):

select *
from 
 ( 
   SELECT      M.Customer_ID
            , M.customer_type

   from        ESRE.MEAS_CUST_TBL as M
   -- must be large enough to still return 2000 rows in the next step
   sample      3000
 ) as M
WHERE       M.Customer_ID is not null
sample 2000

Now you get a fast sample step first returning a small subset of the rows, followed by the filter on NOT NULL and the 2nd sample. Of course, you need some knowledge of the actual data to decide an appropriate sample size, otherwise the outer sample might not return enough rows. But it seems like you just want to examine some rows, in that case you probably don't care if the query returns exactly 2000 rows.