将CSV文件导入到Oracle外部表中

Question

我已经看到各种类似的问题，但是似乎没有一种解决方案适合我。

我得到了在大型机上生成的CSV文件，需要将其加载到Oracle中。 我决定尝试将其映射到Oracle外部表中，然后使用它将其插入Oracle。

这是我的CSV：

CONTRACT_NUMBER,PRODUCTCODE,TRANSACTION_NUMBER,EFFECTIVE_DATE,AMENDMENT,TERM,ACTIVE,AGENT_NUMBER,PREMIUM,ICRATE,RCRATE,IC_ALLOW,RC_ALLOW,SPRATE,TRANSACTION_CODE,TRANSACTION_DATE,AGENT_CATEGORY,AGENT_SALES_CODE,FREQ,TOT_PREMTD,REFERENCE,ALTERNATIVE_COMMISSION_METHOD,PAXUS_REF_ID
PAXUSCT1,MAA,1,07/10/2017,NB,12,Y,2905,6000,,,1,1,,T642,,,,,6000,,,
PAXUSCT1,MAA,2,07/05/2018,INC,11,Y,2905,2400,90,3,1,1,,,,,,,8400,,,
PAXUSCT2,MAA,1,01/06/2018,NB,12,Y,T1000,540,,,1,1,,,,,,,540,,,
PAXUSCT3,MAA,1,05/06/2018,NB,12,Y,T1000,1200,,,1,1,,,,,,,1200,,,

我创建了此定义以及此定义的许多其他变体，但我不断收到错误：

create table LD_CMS_BASIS_MIGRATION
(
  contract_number               VARCHAR2(8),
  productcode                   VARCHAR2(3),
  transaction_number            NUMBER,
  effective_date                DATE,
  amendment                     VARCHAR2(3),
  term                          NUMBER,
  active                        VARCHAR2(1),
  agent_number                  VARCHAR2(5),
  premium                       NUMBER,
  icrate                        NUMBER,
  rcrate                        NUMBER,
  ic_allow                      NUMBER,
  rc_allow                      NUMBER,
  sprate                        NUMBER,
  transaction_code              VARCHAR2(4),
  transaction_date              DATE,
  agent_category                VARCHAR2(4),
  agent_sales_code              VARCHAR2(4),
  freq                          VARCHAR2(1),
  tot_premtd                    NUMBER,
  reference                     VARCHAR2(40),
  alternative_commission_method VARCHAR2(40),
  paxus_ref_id                  VARCHAR2(8)
)
organization external
(
  type ORACLE_LOADER
  default directory MIGRATIONS
  access parameters 
  (
    records field names all files
    fields CSV without embedded record terminators
  )
  location (MIGRATIONS:'CMS_BASIS_MIG.csv')
)
reject limit UNLIMITED;

当我尝试从中读取时，出现此错误：

这是服务器上的日志文件中的内容：

KUP-05004:   Warning: Intra source concurrency disabled because parallel select was not requested.

Field Definitions for table LD_CMS_BASIS_MIGRATION
  Record format DELIMITED BY NEWLINE
  Data in file has same endianness as the platform
  Rows with all null fields are accepted

  Fields in Data Source: 

    CONTRACT_NUMBER                 CHAR (255)
      Terminated by ","
      Enclosed by """ and """
      Trim whitespace same as SQL Loader
    PRODUCTCODE                     CHAR (255)
      Terminated by ","
      Enclosed by """ and """
      Trim whitespace same as SQL Loader
    TRANSACTION_NUMBER              CHAR (255)
      Terminated by ","
      Enclosed by """ and """
      Trim whitespace same as SQL Loader
    EFFECTIVE_DATE                  CHAR (255)
      Terminated by ","
      Enclosed by """ and """
      Trim whitespace same as SQL Loader
    AMENDMENT                       CHAR (255)
      Terminated by ","
      Enclosed by """ and """
      Trim whitespace same as SQL Loader
    TERM                            CHAR (255)
      Terminated by ","
      Enclosed by """ and """
      Trim whitespace same as SQL Loader
    ACTIVE                          CHAR (255)
      Terminated by ","
      Enclosed by """ and """
      Trim whitespace same as SQL Loader
    AGENT_NUMBER                    CHAR (255)
      Terminated by ","
      Enclosed by """ and """
      Trim whitespace same as SQL Loader
    PREMIUM                         CHAR (255)
      Terminated by ","
      Enclosed by """ and """
      Trim whitespace same as SQL Loader
    ICRATE                          CHAR (255)
      Terminated by ","
      Enclosed by """ and """
      Trim whitespace same as SQL Loader
    RCRATE                          CHAR (255)
      Terminated by ","
      Enclosed by """ and """
      Trim whitespace same as SQL Loader
    IC_ALLOW                        CHAR (255)
      Terminated by ","
      Enclosed by """ and """
      Trim whitespace same as SQL Loader
    RC_ALLOW                        CHAR (255)
      Terminated by ","
      Enclosed by """ and """
      Trim whitespace same as SQL Loader
    SPRATE                          CHAR (255)
      Terminated by ","
      Enclosed by """ and """
      Trim whitespace same as SQL Loader
    TRANSACTION_CODE                CHAR (255)
      Terminated by ","
      Enclosed by """ and """
      Trim whitespace same as SQL Loader
    TRANSACTION_DATE                CHAR (255)
      Terminated by ","
      Enclosed by """ and """
      Trim whitespace same as SQL Loader
    AGENT_CATEGORY                  CHAR (255)
      Terminated by ","
      Enclosed by """ and """
      Trim whitespace same as SQL Loader
    AGENT_SALES_CODE                CHAR (255)
      Terminated by ","
      Enclosed by """ and """
      Trim whitespace same as SQL Loader
    FREQ                            CHAR (255)
      Terminated by ","
      Enclosed by """ and """
      Trim whitespace same as SQL Loader
    TOT_PREMTD                      CHAR (255)
      Terminated by ","
      Enclosed by """ and """
      Trim whitespace same as SQL Loader
    REFERENCE                       CHAR (255)
      Terminated by ","
      Enclosed by """ and """
      Trim whitespace same as SQL Loader
    ALTERNATIVE_COMMISSION_METHOD   CHAR (255)
      Terminated by ","
      Enclosed by """ and """
      Trim whitespace same as SQL Loader
    PAXUS_REF_ID                    CHAR (255)
      Terminated by ","
      Enclosed by """ and """
      Trim whitespace same as SQL Loader
KUP-04117: Field name PAXUS_REF_ID
 was not found in the access parameter field list or table.
KUP-04093: error processing the FIELD NAMES record in data file /u02/CAMS/MIGRATIONS/dataload/CMS_BASIS_MIG.csv

任何帮助是极大的赞赏。

谢谢，Mac

##################编辑

下面是塔吉什的回答，差不多。 这是有效的。 我认为换行命令主要是解决问题的方法。 当我编辑了CSV文件并在每行的末尾添加了一个逗号后，它就选择了最后一列。 我也确实必须添加日期掩码。 但是，Tajesh解决方案意味着我不需要编辑CSV文件。

 create table LD_CMS_BASIS_MIGRATION ( contract_number VARCHAR2(8), productcode VARCHAR2(3), transaction_number NUMBER, effective_date DATE, amendment VARCHAR2(3), term NUMBER, active VARCHAR2(1), agent_number VARCHAR2(5), premium NUMBER, icrate NUMBER, rcrate NUMBER, ic_allow NUMBER, rc_allow NUMBER, sprate NUMBER, transaction_code VARCHAR2(4), transaction_date DATE, agent_category VARCHAR2(4), agent_sales_code VARCHAR2(4), freq VARCHAR2(1), tot_premtd NUMBER, reference VARCHAR2(40), alternative_commission_method VARCHAR2(40), paxus_ref_id VARCHAR2(8) ) ORGANIZATION EXTERNAL ( TYPE ORACLE_LOADER DEFAULT DIRECTORY "MIGRATIONS" ACCESS PARAMETERS ( RECORDS DELIMITED BY NEWLINE BADFILE 'CMS_BASIS_MIG_BAD.bad' LOGFILE 'CMS_BASIS_MIG_LOG.log' SKIP 1 FIELDS TERMINATED BY ',' DATE_FORMAT DATE MASK "dd/mm/yyyy" MISSING FIELD VALUES ARE NULL ) LOCATION ( 'CMS_BASIS_MIG.csv' ) ) REJECT LIMIT UNLIMITED PARALLEL 5; 

Answer 1

您可以尝试使用以下创建表语法吗？

create table LD_CMS_BASIS_MIGRATION
(
  contract_number               VARCHAR2(8),
  productcode                   VARCHAR2(3),
  transaction_number            NUMBER,
  effective_date                DATE,
  amendment                     VARCHAR2(3),
  term                          NUMBER,
  active                        VARCHAR2(1),
  agent_number                  VARCHAR2(5),
  premium                       NUMBER,
  icrate                        NUMBER,
  rcrate                        NUMBER,
  ic_allow                      NUMBER,
  rc_allow                      NUMBER,
  sprate                        NUMBER,
  transaction_code              VARCHAR2(4),
  transaction_date              DATE,
  agent_category                VARCHAR2(4),
  agent_sales_code              VARCHAR2(4),
  freq                          VARCHAR2(1),
  tot_premtd                    NUMBER,
  reference                     VARCHAR2(40),
  alternative_commission_method VARCHAR2(40),
  paxus_ref_id                  VARCHAR2(8)
)
ORGANIZATION EXTERNAL ( TYPE ORACLE_LOADER
    DEFAULT DIRECTORY "MIGRATIONS" ACCESS PARAMETERS (
        RECORDS DELIMITED BY NEWLINE
            BADFILE 'CMS_BASIS_MIG_BAD.bad'
            LOGFILE 'CMS_BASIS_MIG_LOG.log'
            SKIP 1
        FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' MISSING FIELD VALUES ARE NULL
    ) LOCATION ( 'CMS_BASIS_MIG.csv' )
) REJECT LIMIT UNLIMITED
    PARALLEL 5;

如果提到的代码引发任何类型的“日期转换”错误，那么如果列的数据类型是日期，则必须指定每个列名称及其格式。 示例： 转换格式

Answer 2

问题是您的csv文件中缺少值。 您需要将它们转换为null。 只需在ACCESS PARAMETERS添加“ MISSING FIELD VALUES ARE NULL

 access parameters 
  (
    records field names all files
    fields CSV without embedded record Terminators
    MISSING FIELD VALUES ARE NULL
  )

Answer 3

由于错误提示“ KUP-04117：在访问参数字段列表或表中找不到字段名PAXUS_REF_ID。”，Oracle无法为列PAXUS_REF_ID找到值，相反，它正在获取新的换行字符，因为没有在记录的最后一个逗号之后为此列填充的值，因此引发错误。

我已经如下修改了CSV文件的第二行和第三行，并且按预期生成了输出。

在第二条记录的末尾添加值0，在第三条记录的末尾添加空格。 更改后，外部表能够读取这两个记录

CONTRACT_NUMBER,PRODUCTCODE,TRANSACTION_NUMBER,EFFECTIVE_DATE,AMENDMENT,TERM,ACTIVE,AGENT_NUMBER,PREMIUM,ICRATE,RCRATE,IC_ALLOW,RC_ALLOW,SPRATE,TRANSACTION_CODE,TRANSACTION_DATE,AGENT_CATEGORY,AGENT_SALES_CODE,FREQ,TOT_PREMTD,REFERENCE,ALTERNATIVE_COMMISSION_METHOD,PAXUS_REF_ID
PAXUSCT1,MAA,1,07/10/2017,NB,12,Y,2905,6000,,,1,1,,T642,,,,,6000,,,0
PAXUSCT1,MAA,2,07/05/2018,INC,11,Y,2905,2400,90,3,1,1,,,,,,,8400,,,
PAXUSCT2,MAA,1,01/06/2018,NB,12,Y,T1000,540,,,1,1,,,,,,,540,,,
PAXUSCT3,MAA,1,05/06/2018,NB,12,Y,T1000,1200,,,1,1,,,,,,,1200,,,

要解决此问题，如@Hotfix所建议，您必须在访问参数中包含以下提及的语句

missing field values are null

另外，如果您在解释日期列数据时遇到任何问题，可以将以下日期格式化程序添加到访问参数中。

date_format date mask "dd/mm/yyyy"

除此之外，您的数据集似乎也对记录4和5的agent_number列值存在问题，对于number列，其值T1000。