I have an internal named stage where json files are stored and from there I want to store them in snowflake table. The structure of destination table is as follows,
file_name (string)
load_date (timestamp)
data (variant)
I am using the following query to move the data from stage to table
copy into tableName (data) from @stagename/filename.json;
But the above query is only populating the data column, what I want is to insert the timestamp and filename too. Any idea what changes I need to make in the query? Thanks
You need to use a COPY
statement with a transformation - documentation here . When you use that method you can query the metadata of the files to get the filename, row number etc - documentation for that here .
Example file filename.json
uploaded to an internal stage called stagename
:
[{"name": "simon"},{"name": "jason"}, {"name": "jessica"}]
Sql to load create and load table:
-- Create example table first with 3 columns
create or replace transient table test_table
(
file_name varchar,
load_date timestamp,
data variant
);
-- Load with transformation:
copy into test_table (file_name, load_date, data) from (
select
metadata$filename,
current_timestamp,
f.$1
from @stagename/filename.json f
)
file_format = (
type = json
strip_outer_array = true
)
force=true
;
Results:
+-------------+-----------------------------+-----------------------+
|FILE_NAME |LOAD_DATE |DATA |
+-------------+-----------------------------+-----------------------+
|filename.json|2021-07-16 08:56:24.075000000|{"name": "simon"} |
|filename.json|2021-07-16 08:56:24.075000000|{"name": "jason"} |
|filename.json|2021-07-16 08:56:24.075000000|{"name": "jessica"} |
+-------------+-----------------------------+-----------------------+
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.