如何使用 Presto SQL 获得可重复的样本？

Question

I am trying to get a sample of data from a large table and want to make sure this can be repeated later on. Other SQL allow repeatable sampling to be done with either setting a seed using set.seed(integer) or repeatable (integer) command. However, this is not working for me in Presto. Is such a command not available yet? Thanks.

Answer 1

One solution is that you can simulate the sampling by adding a column (or create a view) with random stuff (such as UUID) and then selecting rows by filtering on this column (for example, UUID ended with '1'). You can tune the condition to get the sample size you need.

By design, the result is random and also repeatable across multiple runs.

Answer 2

You may create a simple intermediate table with selected ids:

CREATE TABLE IF NOT EXISTS <temp1>
AS
SELECT <id_column>
FROM <tablename> TABLESAMPLE SYSTEM (10);

This will contain only sampled ids and will be ready to use it downstream in your analysis by doing JOIN with data of interest.

Answer 3

If you are using Presto 0.263 or higher you can use key_sampling_percent to reproducibly generate a double between 0.0 and 1.0 from a varchar .

For example, to reproducibly sample 20% of records in table using the id column:

select
    id
from table
where key_sampling_percent(id) < 0.2

If you are using an older version of Presto (eg AWS Athena), you can use what's in the source code for key_sampling_percent :

select
    id
from table
where (abs(from_ieee754_64(xxhash64(cast(id as varbinary)))) % 100) / 100. < 0.2

Answer 4

I have found that you have to use from_big_endian_64 instead of from_ieee754_64 to get reliable results in Athena. Otherwise I got no many numbers close to zero because of the negative exponent.

select id
    from table
    where (abs(from_big_endian_64(xxhash64(cast(id as varbinary)))) % 100) / 100. < 0.2