如何将 SQL 查询的结果导出为 csv 样式的格式化字符串，然后将其保存到 Oracle 中的 clob 中？

Question

我有以下问题：我有一个表 Source_Data（实际上，有很多不同的源表），我需要将它们中的一些数据导出到 Result_Table 中。 对于每个源表，我都有一些返回数据的 sql，需要将其转换为类似 csv 的字符串。

例如，Source_Data1 表具有 PK - ID、一些外键 - A_ID、B_ID、CREATION_DATE 日期和一些其他列。

我需要导出从 Source_Data1 表中选择数据的 SQL 查询的结果，它通常返回主键、一些非空的外键和一个也不为空的日期，并将其转换为类似 cvs 的字符串，这应该稍后作为clob保存在其他表中。

CREATE TABLE Source_Data1 (
    ID NUMBER(3) NOT NULL,
    A_ID NUMBER(10) NOT NULL, 
    B_ID NUMBER(10) NOT NULL, 
    CREATION_DATE DATE NOT NULL, 
    some other columns );

我有 SQL 查询：

select ID, A_ID, B_ID, CREATION_DATE
from Source_Data1
where <expression>

基于这个查询，我需要生成这样的字符串：

'ID,A_ID,B_ID,CREATION_DATE
1,200,10,2018-03-01 00:00:00.0
7,202,11,2018-03-02 00:00:00.0
8,205,11,2018-03-02 00:00:00.0'

我需要将它作为clob保存到结果表中：

Insert into Result_Table (ID, SOURCE_NAME, DATA) values         
(result_seq.nextval, 'Source_Data1', result of sqlquery 1);

很多源表都需要这样做。

我想知道是否有一些通用的方法来做到这一点？ 因为对于每个源表，我都有自己的 sql 查询，不同的选定列集，手动生成这个 csv 字符串看起来不太好。

感谢您的意见。

Answer 1

第 1 步应该是下载SQLcl ，您可以通过“SET SQLFORMAT CSV”和 SPOOL 轻松地将 SQL select 语句的输出保存为 CSV 格式，它为您完成工作。 因此，您可以使用它来遍历一堆表名/选择语句，通过 shell 脚本将每个语句的输出保存到一个目录中。

接下来，这是一个包含一堆文件实用程序的包，这些实用程序都与 oracle DIRECTORY 对象和其中的文件一起使用。 使用这个和一些 PL/SQL，您可以轻松地将那些已保存的文件提取到表中。 可能有更简单的方法来做你想做的事情，如果有，我期待为其他人听到它们。 这是一个需要解决的大问题。

1) file_into_blob - 将文件放入数据库
2) convert_blob_to_clob - 转换为clob
3）然后您可以将其插入到您的表中。

包装规格

Create or replace package fileutils as

   --
   -- This procedure deletes a file, and depends on an Oracle DIRECTORY object being passed
   --
   Procedure delete_os_file (i_directory varchar2, i_filename varchar2);

   --
   -- This procedure moves and optionally renames a file, 
   -- and depends on an Oracle DIRECTORY object being passed
   --
   Procedure move_os_file ( i_source_directory in varchar2, i_source_file in varchar2, i_target_directory in varchar2, i_target_file in varchar2);

   --
   -- This procedure takes a blob variable and writes it to a file, 
   -- and depends on an Oracle DIRECTORY object being passed
   --
   Procedure blob_into_file (i_directory in varchar2, i_file_name in varchar2, i_blob in blob);

   --
   -- This procedure takes a file and uploads it into a blob variable
   -- and depends on an Oracle DIRECTORY object being passed
   --
   Procedure file_into_blob(i_directory in varchar2, i_file_name in varchar2, o_blob out blob);

   --
   -- This procedure converts a clob to a blob
   --
   Procedure convert_clob_to_blob (i_clob in clob, o_blob out blob);

   --
   -- This procedure converts a blob to a clob
   --
   Procedure convert_blob_to_clob (i_blob in blob, o_clob out clob);

   --
   -- This one checks for file existence without Java
   --
   Function file_exists (i_directory in varchar2, i_filename in varchar2) return boolean;

   --
   -- Returns the basename of a filename
   -- Works with Windows and UNIX pathnames
   --
   Function basename (i_filename in varchar2) return varchar2;

   --
   -- This takes a Base64 string and converts it to a binary BLOB
   --
   Procedure base64_string_to_blob (i_clob in clob, o_blob out blob);
   Function base64_string_to_blob (i_clob in clob) return blob;

   --
   -- This takes a binary BLOB and converts it to a Base64 string
   --
   Procedure blob_to_base64_string (i_blob in blob, o_clob out clob);
   Function blob_to_base64_string (i_blob in blob) return clob;

End fileutils;
/

Show error;

包体

Set define off;

Create or replace package body fileutils as

   Procedure delete_os_file (i_directory varchar2, i_filename varchar2)
   is

   Begin

      utl_file.fremove(i_directory,i_filename);

   End;

   Procedure move_os_file
   (
      i_source_directory     in varchar2,
      i_source_file          in varchar2,
      i_target_directory     in varchar2,
      i_target_file          in varchar2
   )

   is

      srcdir               varchar2(255) := upper(i_source_directory);
      tgtdir               varchar2(255) := upper(i_target_directory);

   begin

      --
      -- NOTE: If you're getting the all-too-familiar
      -- ORA-29292: file rename operation failed
      -- and you're SURE that your directory names are correct,
      -- and you're SURE that your privileges are correct, both at the
      -- OS level, and within the database, there's one last thing that
      -- can get you. I learned the hard way that this command will NOT
      -- work successfully renaming a file from one filesystem to another,
      -- at least when those filesystems are NFS mounted. That is all.
      --

      utl_file.frename(srcdir,i_source_file,tgtdir,i_target_file,TRUE);

   end move_os_file;

   Procedure blob_into_file (i_directory in varchar2, i_file_name in varchar2, i_blob in blob)
   is

      l_file            utl_file.file_type;
      l_buffer          raw(32767);
      l_amount          binary_integer := 32767;
      l_pos             integer := 1;
      i_blob_len        integer;

   Begin

      i_blob_len := dbms_lob.getlength(i_blob);
      l_pos:= 1;

      -- Open the destination file.
      l_file := utl_file.fopen(i_directory,i_file_name,'wb', 32767);

      -- Read chunks of the BLOB and write them to the file
      -- until complete.
      while l_pos < i_blob_len loop
         dbms_lob.read(i_blob, l_amount, l_pos, l_buffer);
         utl_file.put_raw(l_file, l_buffer, TRUE);
         l_pos := l_pos + l_amount;
      end loop;

      -- Close the file.
      utl_file.fclose(l_file);

   End blob_into_file;

   Procedure file_into_blob(i_directory in varchar2, i_file_name in varchar2, o_blob out blob) 
   is
      src_loc       bfile   := bfilename(i_directory, i_file_name);
   Begin

      -- Initialize the dest blob
      o_blob := empty_blob();

      -- Open source binary file from OS
      dbms_lob.open(src_loc, dbms_lob.lob_readonly);

      -- Create temporary LOB object
      dbms_lob.createtemporary(
            lob_loc => o_blob
          , cache   => true
          , dur     => dbms_lob.session
      );

      -- Open temporary lob
      dbms_lob.open(o_blob, dbms_lob.lob_readwrite);

      -- Load binary file into temporary LOB
      dbms_lob.loadfromfile(
            dest_lob => o_blob
          , src_lob  => src_loc
          , amount   => dbms_lob.getLength(src_loc));

      -- Close lob objects
      dbms_lob.close(o_blob);
      dbms_lob.close(src_loc);

   End file_into_blob;

   Function basename (i_filename in varchar2) return varchar2
   is
      v_basename        varchar2(1024);
   Begin

      --
      -- If the regex's below don't match, then it's already at its base name
      -- Return what was passed.
      --
      v_basename := i_filename;

      if regexp_like(i_filename,'^.*\\') then
         dbms_output.put_line('This is a Windows file');
         v_basename := regexp_substr(i_filename,'[^\]*$');
         dbms_output.put_line('Basename is : '||v_basename);
      end if;
      if regexp_like(i_filename,'^/') then
         dbms_output.put_line('This is a UNIX file');
         v_basename := regexp_substr(i_filename,'[^/]*$');
         dbms_output.put_line('Basename is : '||v_basename);
      end if;

      return v_basename;

   End basename;

   Function file_exists (i_directory in varchar2, i_filename in varchar2) return boolean
   is
      v_exists          boolean;
      v_file_length     number;
      v_block_size      number;
   Begin
      utl_file.fgetattr(upper(i_directory), i_filename, v_exists, v_file_length, v_block_size);   
      if (v_exists) then
         dbms_output.put_line('File '||i_filename||' exists, '||v_file_length||' bytes');
      else
         dbms_output.put_line('File '||i_filename||' does not exist');
      end if;

      return v_exists;

   end file_exists;

   Procedure convert_clob_to_blob (i_clob in clob, o_blob out blob)
   is

      v_in      pls_Integer := 1;
      v_out     pls_Integer := 1;
      v_lang    pls_Integer := 0;
      v_warning pls_Integer := 0;

   Begin

      dbms_lob.createtemporary(o_blob,TRUE);
      dbms_lob.converttoblob(o_blob,i_clob,DBMS_lob.getlength(i_clob),v_in,v_out,dbms_lob.default_csid,v_lang,v_warning);

   End convert_clob_to_blob;

   Procedure convert_blob_to_clob (i_blob in blob, o_clob out clob)
   is

      v_in      pls_Integer := 1;
      v_out     pls_Integer := 1;
      v_lang    pls_Integer := 0;
      v_warning pls_Integer := 0;

   Begin

      dbms_lob.createtemporary(o_clob,TRUE);
      dbms_lob.converttoclob(o_clob,i_blob,DBMS_lob.getlength(i_blob),v_in,v_out,dbms_lob.default_csid,v_lang,v_warning);

   End convert_blob_to_clob;

   Procedure blob_to_base64_string (i_blob in blob, o_clob out clob)
   is

      v_out_cl     clob;
      file_len     pls_integer;
      modulo       pls_integer;
      pieces       pls_integer;
      amt          binary_integer      := 23808;
      buf          raw (32767);
      buf_tx       varchar2(32767);
      pos          pls_integer         := 1;
      filepos      pls_integer         := 1;
      counter      pls_integer         := 1;
   Begin
      dbms_lob.createtemporary (v_out_cl, true, dbms_lob.call);
      file_len := dbms_lob.getlength (i_blob);
      modulo := mod (file_len, amt);
      pieces := trunc (file_len / amt);

      while (counter <= pieces) loop
         dbms_lob.read (i_blob, amt, filepos, buf);
         buf_tx:=utl_raw.cast_to_varchar2 (utl_encode.base64_encode (buf));
         dbms_lob.writeappend (v_out_cl,length(buf_tx),buf_tx);
         filepos := counter * amt + 1;
         counter := counter + 1;
      end loop;

      if (modulo <> 0) THEN
         dbms_lob.read (i_blob, modulo, filepos, buf);
         buf_tx:=utl_raw.cast_to_varchar2 (utl_encode.base64_encode (buf));
         dbms_lob.writeappend (v_out_cl,length(buf_tx),buf_tx);
      end if;

      o_clob := v_out_cl;

   End blob_to_base64_string;

   Function blob_to_base64_string (i_blob in blob) return clob
   is
      v_out_cl     clob;
      file_len     pls_integer;
      modulo       pls_integer;
      pieces       pls_integer;
      amt          binary_integer      := 23808;
      buf          raw (32767);
      buf_tx       varchar2(32767);
      pos          pls_integer         := 1;
      filepos      pls_integer         := 1;
      counter      pls_integer         := 1;
   Begin

      dbms_lob.createtemporary (v_out_cl, true, dbms_lob.call);
      file_len := dbms_lob.getlength (i_blob);
      modulo := mod (file_len, amt);
      pieces := trunc (file_len / amt);

      while (counter <= pieces) loop
         dbms_lob.read (i_blob, amt, filepos, buf);
         buf_tx:=utl_raw.cast_to_varchar2 (utl_encode.base64_encode (buf));
         dbms_lob.writeappend (v_out_cl,length(buf_tx),buf_tx);
         filepos := counter * amt + 1;
         counter := counter + 1;
      end loop;

      if (modulo <> 0) THEN
         dbms_lob.read (i_blob, modulo, filepos, buf);
         buf_tx:=utl_raw.cast_to_varchar2 (utl_encode.base64_encode (buf));
         dbms_lob.writeappend (v_out_cl,length(buf_tx),buf_tx);
      end if;

      return v_out_cl;

   End blob_to_base64_string;

   Procedure base64_string_to_blob (i_clob in clob, o_blob out blob)
   is

      v_out_bl blob;
      clob_size number;
      pos number;
      charBuff varchar2(32767);
      dBuffer RAW(32767);
      v_readSize_nr number;
      v_line_nr number;

   begin
      dbms_lob.createTemporary (v_out_bl, true, dbms_lob.call);
      v_line_nr:=greatest(65, instr(i_clob,chr(10)), instr(i_clob,chr(13)));
      v_readSize_nr:= floor(32767/v_line_nr)*v_line_nr;
      clob_size := dbms_lob.getLength(i_clob);
      pos := 1;

      while (pos < clob_size) loop
         dbms_lob.read (i_clob, v_readSize_nr, pos, charBuff);
         dBuffer := UTL_ENCODE.base64_decode (utl_raw.cast_to_raw(charBuff));
         dbms_lob.writeAppend (v_out_bl,utl_raw.length(dBuffer),dBuffer);
         pos := pos + v_readSize_nr;
      end loop;

      o_blob := v_out_bl;

   end base64_string_to_blob;

   Function  base64_string_to_blob (i_clob in clob) return blob
   is

      v_out_bl blob;
      clob_size number;
      pos number;
      charBuff varchar2(32767);
      dBuffer RAW(32767);
      v_readSize_nr number;
      v_line_nr number;

   begin
      dbms_lob.createTemporary (v_out_bl, true, dbms_lob.call);
      v_line_nr:=greatest(65, instr(i_clob,chr(10)), instr(i_clob,chr(13)));
      v_readSize_nr:= floor(32767/v_line_nr)*v_line_nr;
      clob_size := dbms_lob.getLength(i_clob);
      pos := 1;

      while (pos < clob_size) loop
         dbms_lob.read (i_clob, v_readSize_nr, pos, charBuff);
         dBuffer := UTL_ENCODE.base64_decode (utl_raw.cast_to_raw(charBuff));
         dbms_lob.writeAppend (v_out_bl,utl_raw.length(dBuffer),dBuffer);
         pos := pos + v_readSize_nr;
      end loop;

      return v_out_bl;

   end base64_string_to_blob;

end fileutils;
/

Show error;

Answer 2

我想你可以使用 LISTAGG 功能

select listagg(ONE_LINE) WITHIN GROUP (ORDER BY ROW_NUM)
  from (
        select 'ID,A_ID,B_ID,CREATION_DATE' || CHR(10) as ONE_LINE,
               -1 as ROW_NUM
          from dual
         union all
        select ID ||','|| A_ID ||','|| B_ID ||','|| CREATION_DATE || CHR(10) as ONE_LINE,
               ROWNUM as ROW_NUM
        from Source_Data1
        where <expression>
       );

Answer 3

以下代码将使用DATA_DUMP.SQL和函数DBMS_XSLPROCESSOR.READ2CLOB将表的 CSV 版本创建为 CLOB 行。

首先，在您的架构上安装DATA_PUMP过程。 该程序是单个文件中的单个命令，您可以通过 SQL*Plus 运行它，也可以将其复制并粘贴到您的 IDE 中。

接下来，在服务器上创建一个目录来临时存储文件。 大多数现有实用程序旨在创建文件。 写入文件并将其作为 CLOB 读取比修改实用程序以直接写入表更容易。

create or replace directory temp_dir as 'C:\temp';

然后这个 PL/SQL 块将读取和写入数据：

--Store a table as a single CSV clob.
--TODO: Delete the files from the server when done.
declare
    v_clob clob;
begin
    for tables in
    (
        --Query that returns the table names.
        --(Doesn't have to be hard-coded, maybe a query on DBA_TABLES would help?)
        select column_value table_name
        from table(sys.odcivarchar2list('source_data'))
    ) loop
        data_dump
        (
            query_in        => 'select * from source_data1',
            file_in         => 'source_data1.csv',
            directory_in    => 'temp_dir',
            nls_date_fmt_in => 'YYYY-MM-DD HH24:MI:SS',
            delimiter_in    => ',',
            header_row_in    => true
        );

        v_clob := dbms_xslprocessor.read2clob
        (
            flocation => 'TEMP_DIR',
            fname     => 'source_data1.csv'
        );

        insert into result_table (id, source_name, data) values         
        (result_seq.nextval, 'source_data1', v_clob);

        --I wouldn't normally commit after each row, but when exporting large
        --tables the script may run too long and there's a good chance of an
        --error, so partial results may be helpful.
        commit;
    end loop;
end;
/

读取和写入 CSV 文件不是火箭科学，但并不像大多数人想象的那么简单。 99% 的 CSV 程序无法处理数据中的逗号或添加标题行之类的问题。 因此，最好查找预先存在的代码，而不是编写自己的实用程序。