![](/img/trans.png)
[英]Spark SQL(Hive query through HiveContext) always creating 31 partitions
[英]Spark SQL(Hive query through HiveContext) INSERT OVERWRITE is not overwriting existing data if multiple partition is present in hive table
//Hive-1.2.1000.2.6.1.0-129我們試圖插入具有多個分區的test5表。根據document( https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML )INSERT OVERWRITE將覆蓋表或分區中的所有現有數據。 但是,在觸發INSERT OVERWRITE查詢之后,我們仍然會得到一些舊數據。 下面是示例執行和輸出。
//Spark-2.1.1在Spark-2.1.1中通過HiveContext運行時我們得到的結果相同
CREATE TABLE dbtest.test5 (emp_id INT) PARTITIONED BY (depart_id INT,depart_name STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' STORED AS TEXTFILE LOCATION 'externalpath';
INSERT INTO TABLE dbtest.test5 PARTITION (depart_id,depart_name) SELECT emp_id,depart_id,depart_name from dbtest.tempTableHive1;
4 123 Dev
5 123 Dev
6 123 Test
7 567 Test
INSERT INTO TABLE dbtest.test5 PARTITION (depart_id,depart_name) SELECT emp_id,depart_id,depart_name from dbtest.tempTableHive2;
4 123 Dev
5 123 Dev
1 123 Dev
2 123 Dev
6 123 Test
3 123 Test
7 567 Test
INSERT OVERWRITE TABLE dbtest.test5 PARTITION (depart_id,depart_name) SELECT emp_id,depart_id,depart_name from dbtest.tempTableHive3;
8 123 Dev
9 123 Dev
10 123 Dev
6 123 Test
3 123 Test
7 567 Test
代碼是否有錯誤,或者是Apache配置單元問題?
當您指定INSERT OVERWRITE時,Hive將覆蓋分區。 請參閱下面的cloudera快速啟動VM的輸出。
hive> SELECT * FROM tempTableHive1;
OK
4 123 Dev
5 567 Test
Time taken: 0.048 seconds, Fetched: 2 row(s)
hive> INSERT INTO TABLE test5 PARTITION (depart_id,depart_name) SELECT emp_id,depart_id,depart_name from tempTableHive1;
hive> SELECT * FROM test5;
OK
4 123 Dev
5 567 Test
Time taken: 0.065 seconds, Fetched: 2 row(s)
hive> SELECT * FROM tempTableHive2;
OK
4 123 Dev
6 123 Dev
Time taken: 0.047 seconds, Fetched: 2 row(s)
hive> INSERT INTO TABLE test5 PARTITION (depart_id,depart_name)
> SELECT emp_id,depart_id,depart_name from tempTableHive2;
hive> SELECT * FROM test5;
OK
4 123 Dev
4 123 Dev
6 123 Dev
5 567 Test
Time taken: 0.057 seconds, Fetched: 4 row(s)
hive> SELECT * FROM tempTableHive3;
OK
100 123 Dev
101 123 Dev
hive> INSERT OVERWRITE TABLE test5 PARTITION (depart_id,depart_name)
> SELECT emp_id,depart_id,depart_name from tempTableHive3;
hive> SELECT * FROM test5;
OK
100 123 Dev
101 123 Dev
5 567 Test
Time taken: 0.072 seconds, Fetched: 3 row(s)
如果仍然遇到問題,調試此問題的最佳方法是檢查HDFS文件。 每個部門ID /部門名稱組合應只有一個文件。 示例/ user / hive / warehouse / test5 / depart_id = 123 / depart_name = Dev。 由於它們是文本文件,因此您可以快速“捕獲”它們以查看內容。 讓我們知道你過得如何。
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.