將 JSON 數據從 Snowflake 復制到 S3

Question

我正在嘗試將數據（變體數據類型 - JSON）從雪花復制到 S3。 我正在使用以下命令：

copy into @STG_SF_S3_DEV_JJC/sample_file1.json
from (select distinct RECORD_CONTENT from MYTABLE where LOAD_DT_TS >= '2022-02-09 00:00:00')
FILE_FORMAT = (TYPE=JSON,COMPRESSION = NONE)
SINGLE=true
header=true
OVERWRITE = TRUE;

數據正在被復制。 我正在使用 STORAGE_INTEGRATION 進程並創建 STAGE 表。 我的問題是：

1. 對於上面的查詢，我得到 23 行，我想在 S3 中將這 23 行加載為 23 個單獨的文件（JSON 文件）。 如何做到這一點？
2. 每條記錄都有不同的 LOAD_DT_TS，因此對於每 23 個文件，我需要給出不同的名稱。 我的意思是，文件名應該以 LOAD_DT_TS 結尾。 如何做到這一點？
3. MYTABLE在上面的查詢（COPY 命令）中，我有 4 列。 有沒有可能將所有 4 列加載到單個 JSON 文件中的方法？

請分享你的想法。

Answer 1

請參閱下文（它涵蓋了第 1 點和第 2 點）。

SNOWFLAKE1#COMPUTE_WH@TEST_DB.PUBLIC>select * from test_pivot;
+---------+-----------+-----------+-----------+
| COMPANY | BU        | EVAL_DESC | MEMBER_ID |
|---------+-----------+-----------+-----------|
| C1      | FINANCIAL | L1        | ID1       |
| C1      | FINANCIAL | L2        | ID2       |
| C1      | FINANCIAL | L3        | ID3       |
| C1      | HR        | L1        | ID4       |
| C1      | HR        | L2        | ID5       |
| C2      | FINANCIAL | L1        | ID6       |
| C2      | BUSINESS  | L1        | ID7       |
+---------+-----------+-----------+-----------+
7 Row(s) produced. Time Elapsed: 0.187s
SNOWFLAKE1#COMPUTE_WH@TEST_DB.PUBLIC>list @test_row_stage;
+------+------+-----+---------------+
| name | size | md5 | last_modified |
|------+------+-----+---------------|
+------+------+-----+---------------+
0 Row(s) produced. Time Elapsed: 0.177s
SNOWFLAKE1#COMPUTE_WH@TEST_DB.PUBLIC>EXECUTE IMMEDIATE $$
                                     DECLARE
                                       company varchar2(30);
                                       BU varchar2(30);
                                       eval_desc varchar2(30);
                                       member_id varchar2(30);
                                       file_name varchar2(30);
                                       c1 CURSOR FOR SELECT * FROM test_pivot;
                                     BEGIN
                                      // OPEN c1;
                                       for record in c1 do
                                       company:=record.company;
                                       BU:=record.BU;
                                       eval_desc:=record.eval_desc;
                                       member_id:=record.member_id;
                                       file_name:='load'||'_'||member_id||'.csv';
                                       create or replace temporary table temp_test_pvt(company varchar2(30),BU varchar2
                                     (30),eval_desc varchar2(30),member_id varchar2(30));
                                       insert into temp_test_pvt values (:company,:bu,:eval_desc,:member_id);
                                       execute immediate 'copy into @test_row_stage/'||:file_name||' from (select * fro
                                     m temp_test_pvt) overwrite=false';
                                       end for;
                                       RETURN 0;
                                     END;
                                     $$
                                     ;
+-----------------+
| anonymous block |
|-----------------|
|               0 |
+-----------------+
1 Row(s) produced. Time Elapsed: 9.803s
SNOWFLAKE1#COMPUTE_WH@TEST_DB.PUBLIC>list @test_row_stage;
+------------------------------------------+------+----------------------------------+-------------------------------+
| name                                     | size | md5                              | last_modified                 |
|------------------------------------------+------+----------------------------------+-------------------------------|
| test_row_stage/load_ID1.csv_0_0_0.csv.gz |   48 | df314a0d95a771d5d81fa9b1cfb3a28e | Thu, 17 Mar 2022 16:47:42 GMT |
| test_row_stage/load_ID2.csv_0_0_0.csv.gz |   48 | 6be39868046f583b1c63d616faa9e7f6 | Thu, 17 Mar 2022 16:47:43 GMT |
| test_row_stage/load_ID3.csv_0_0_0.csv.gz |   48 | ecf9dbcb4e45fa29d6bcfe268c6ccae3 | Thu, 17 Mar 2022 16:47:44 GMT |
| test_row_stage/load_ID4.csv_0_0_0.csv.gz |   48 | 0fd3ff7e0a453e04be3aca22147a7d32 | Thu, 17 Mar 2022 16:47:45 GMT |
| test_row_stage/load_ID5.csv_0_0_0.csv.gz |   48 | c77985f8312a540816d82b4bf4ec5249 | Thu, 17 Mar 2022 16:47:46 GMT |
| test_row_stage/load_ID6.csv_0_0_0.csv.gz |   48 | c9e9d6fd613a8bdb76413dd3e9464cc4 | Thu, 17 Mar 2022 16:47:47 GMT |
| test_row_stage/load_ID7.csv_0_0_0.csv.gz |   48 | 4e4b999ed56059b44ee6bd15f28cafb8 | Thu, 17 Mar 2022 16:47:48 GMT |
+------------------------------------------+------+----------------------------------+-------------------------------+
7 Row(s) produced. Time Elapsed: 0.176s
SNOWFLAKE1#COMPUTE_WH@TEST_DB.PUBLIC>select $1,$2,$3,$4 from @test_row_stage/load_ID1.csv_0_0_0.csv.gz;
+----+-----------+----+-----+
| $1 | $2        | $3 | $4  |
|----+-----------+----+-----|
| C1 | FINANCIAL | L1 | ID1 |
+----+-----------+----+-----+
1 Row(s) produced. Time Elapsed: 0.429s
SNOWFLAKE1#COMPUTE_WH@TEST_DB.PUBLIC>select $1,$2,$3,$4 from @test_row_stage/load_ID2.csv_0_0_0.csv.gz;
+----+-----------+----+-----+
| $1 | $2        | $3 | $4  |
|----+-----------+----+-----|
| C1 | FINANCIAL | L2 | ID2 |
+----+-----------+----+-----+
1 Row(s) produced. Time Elapsed: 0.374s
SNOWFLAKE1#COMPUTE_WH@TEST_DB.PUBLIC>select $1,$2,$3,$4 from @test_row_stage/load_ID3.csv_0_0_0.csv.gz;
+----+-----------+----+-----+
| $1 | $2        | $3 | $4  |
|----+-----------+----+-----|
| C1 | FINANCIAL | L3 | ID3 |
+----+-----------+----+-----+
1 Row(s) produced. Time Elapsed: 0.506s
SNOWFLAKE1#COMPUTE_WH@TEST_DB.PUBLIC>select $1,$2,$3,$4 from @test_row_stage/load_ID4.csv_0_0_0.csv.gz;
+----+----+----+-----+
| $1 | $2 | $3 | $4  |
|----+----+----+-----|
| C1 | HR | L1 | ID4 |
+----+----+----+-----+
1 Row(s) produced. Time Elapsed: 0.281s
SNOWFLAKE1#COMPUTE_WH@TEST_DB.PUBLIC>

Answer 2

這適用於輸入帶有引號放置的謂詞值 -

SNOWFLAKE1#COMPUTE_WH@TEST_DB.PUBLIC>EXECUTE IMMEDIATE $$
                                     DECLARE
                                      load_dt_tss timestamp;
                                      file_name varchar2(30);
                                     BEGIN
                                      file_name:='somefile'||'.csv';
                                      execute immediate 'copy into @test_row_stage/'||:file_name||' from (select LOAD_D
                                     T_TS from TEST_TS where LOAD_DT_TS >=' || '''2022-02-09 00:00:00''' || ')' || ' FI
                                     LE_FORMAT = (TYPE=CSV,COMPRESSION = NONE) overwrite=FALSE';
                                      RETURN 0;
                                     END;
                                     $$
                                     ;
+-----------------+
| anonymous block |
|-----------------|
|               0 |
+-----------------+
1 Row(s) produced. Time Elapsed: 0.584s

Answer 3

SINGLE=true - 此選項將完整的 SQL 輸出發送到單個文件，如果您刪除它，它會將輸出發送到單獨的文件。