Hive-動態分區錯誤中面臨的挑戰

Question

任何人都可以指導我在進行動態分區時哪里出錯了。

-登台表：

    create table staging_peopledata
    (
    firstname string, 
    secondname string, 
    salary float, 
    country string
    state string
    ) 
    row format delimited fields terminated by ',' lines terminated by '\n';

-登台表的數據：

    John,David,30000,RUS,tnRUS
    John,David,30000,RUS,tnRUS
    Mary,David,5000,AUS,syAUS
    Mary,David,5000,AUS,syAUS
    Mary,David,5000,AUS,weAUS
    Pierre,Cathey,6000,RUS,kaRUS
    Pierre,Cathey,6000,RUS,kaRUS
    Ahmed,Talib,11000,US,bcUS
    Ahmed,Talib,11000,US,onUS
    Ahmed,Talib,11000,US,onUS
    kris,David,80000,UK,lnUK
    kris,David,80000,UK,soUK

-生產表：

    create table Production_peopledata
    (
    firstname string, 
    lastname string, 
    salary float) 
    partitioned by (country string, state string) 
    row format delimited fields terminated by ',' lines terminated by '\n';

    SET hive.exec.dynamic.partition=true;
    SET hive.exec.dynamic.partition.mode=nonstrict;

    insert overwrite table Production_peopledata 
    partition(country,state) 
    select firstname, secondname, salary, country, state from staging_peopledata;

如果我執行上述命令，則會收到如下錯誤。

    FAILED: SemanticException [Error 10096]: Dynamic partition strict mode
     requires atleast one static partition column. To turn this off set
     hive.exec.dynamic.partition.mode=nonstrict

誰能告訴我我在哪里做錯了。

Answer 1

您能在Hive Shell上在命令下方運行嗎？

hive>set hive.exec.dynamic.partition.mode=nonstrict;

Answer 2

您需要設置以下屬性：

set hive.exec.dynamic.partition=true;  
set hive.exec.dynamic.partition.mode=nonstrict;  

要分區的列名不應成為表定義的一部分。 由於分區列是動態生成的。 在將數據填充到分區表中時，分區列應來自源表。

假設我們有EMP和EMP1表。 EMP1是分區表，它將從EMP表獲取數據。 最初，這兩個表都是相同的。 因此，首先我們需要創建一個分區列，即salpart 。 然后，我們將此列添加到源表EMP 。 成功運行后，我們可以在用戶/配置單元/倉庫位置看到分區的文件。 上面的解釋實現如下：

load data local inpath '/home/cloudera/myemployeedata.txt' overwrite into table emp;

CREATE TABLE IF NOT EXISTS emp ( eid int, name String,
salary String, destination String,salpart string)
COMMENT "Employee details"
ROW FORMAT DELIMITED
FIELDS TERMINATED BY "\t"
LINES TERMINATED BY "\n"
STORED AS TEXTFILE;

CREATE TABLE IF NOT EXISTS emp1 ( eid int, name String,
salary String, destination String)
COMMENT "Employee details"
partitioned by (salpart string)  {this column will values will come from a seperate table }
ROW FORMAT DELIMITED
FIELDS TERMINATED BY "\t"
LINES TERMINATED BY "\n"
STORED AS TEXTFILE;

Dynamic Partition:
set hive.exec.dynamic.partition=true;  
set hive.exec.dynamic.partition.mode=nonstrict;  

insert overwrite table emp1 partition(salpart) select eid,name,salary,destination,salpart from emp;

Answer 3

根據錯誤，似乎模式仍然嚴格，對於動態分區，需要在以下命令中將其設置為非嚴格使用

hive>設置hive.exec.dynamic.partition.mode = nonstrict;

Answer 4

再次嘗試執行set hive.exec.dynamic.partition.mode = nonstrict有時在蜂巢中，即使您設置了此屬性，它也會視為嚴格模式，因此它會發生，因此，我建議您再次設置此屬性