[英]Hive: Cannot insert arrays and maps from file in hive table
這是我的表的架構
CREATE DATABASE IF NOT EXISTS mydb;
USE mydb;
CREATE TABLE IF NOT EXISTS mytab (
idcol string,
arrcol array<string>,
mapcol map<string,string>
)
PARTITIONED BY (data_date string)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '|'
STORED AS TEXTFILE;
現在我要做的就是在此表中插入一行。 我在psv文件中有該行
123|["a","b"]|{"1":"a","2":"b"}
這是我嘗試加載數據的方式
USE mydb; LOAD DATA INPATH '/path/to/file' INTO TABLE mytab PARTITION (data_date='2019-02-02');
查詢成功,但是當我看到結果時
hive -e "use mydb; select * from mytab where data_date='2019-02-02';"
我得到
hive> select * from mytab;
OK
123 ["[\"a\",\"b\"]"] {"{\"1\":\"a\",\"2\":\"b\"}":null} 2019-02-02
Time taken: 2.39 seconds, Fetched: 1 row(s)
因此,看起來LOAD
對數據進行了一些轉換。 它使字符串值保持良好,但數組和映射存在一些問題。
如何正確插入數組和映射?
我也嘗試了以下作為輸入
123|array("a","b")|{"1":"a","2":"b"}
加載成功,但是當我查詢數據時,我得到了
root@0d2b0044b4c1:/opt# hive -e "use mydb;select * from mytab;"
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Logging initialized using configuration in jar:file:/opt/hive/lib/hive-common-2.3.2.jar!/hive-log4j2.properties Async: true
OK
Time taken: 6.096 seconds
OK
123 ["array(\"a\",\"b\")"] {"{\"1\":\"a\",\"2\":\"b\"}":null} 1554090675
Time taken: 3.266 seconds, Fetched: 1 row(s)
更新
非常感謝@pedram bashiri的回答。 我創建了外部表並能夠填充它。 但是,所有內容都填充為字符串
hive> drop table if exists extab;
OK
Time taken: 0.01 seconds
hive> create external table extab(idcol string,arrcol array<string>,mapcol map<string,string>, data_date string)
> row format serde 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
> with serdeproperties (
> "separatorChar" = "|",
> "quoteChar" = "\"",
> "escapeChar" = "\\"
> )
> stored as textfile
> location '/tmp/serdes/';
OK
Time taken: 0.078 seconds
hive> desc extab;
OK
idcol string from deserializer
arrcol string from deserializer
mapcol string from deserializer
data_date string from deserializer
Time taken: 0.059 seconds, Fetched: 4 row(s)
hive> select * from extab;
OK
123 ["a","b"] {"1":"a","2":"b"} 2019
Time taken: 0.153 seconds, Fetched: 1 row(s)
hive>
這是存儲在hdfs中的內容
root@0d2b0044b4c1:/opt# hadoop fs -ls -R /tmp/serdes/
-rw-r--r-- 1 root root 37 2019-04-04 22:06 /tmp/serdes/x.psv
root@0d2b0044b4c1:/opt# hadoop fs -cat /tmp/serdes/x.psv
123|["a","b"]|{"1":"a","2":"b"}|2019
root@0d2b0044b4c1:/opt#
我也試過
create external table extab(idcol string,arrcol array<string>,mapcol map<string,string>, data_date string)
row format serde 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
with serdeproperties (
"separatorChar" = "|"
)
stored as textfile
location '/tmp/serdes/';
但仍然,所有內容都存儲為字符串,所以現在當我嘗試插入時,類型不匹配。
使用opencsv根據您的psv文件創建一個外部表,並將其命名為mytab_exterrnal。 指定serdeproperties,例如
with serdeproperties (
"separatorChar" = "|",
"quoteChar" = """,
"escapeChar" = "\\"
)
然后簡單地做
INSERT INTO mytab
SELECT * FROM mytab_external;
https://community.hortonworks.com/articles/8313/apache-hive-csv-serde-example.html
因此,經過大量的挖掘,我發現了
CREATE TABLE IF NOT EXISTS mytab (
idcol string,
arrcol array<string>,
mapcol map<string,string>
)
PARTITIONED BY (data_date string)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '|'
COLLECTION ITEMS TERMINATED BY ','
MAP KEYS TERMINATED BY '='
STORED AS TEXTFILE;
然后,我可以加載以下內容
123|a,b|1=a,2=b|2019
root@0d2b0044b4c1:/opt# hive -e "use mydb; LOAD DATA INPATH '/path/to/file' INTO TABLE mytab PARTITION (data_date='2019');"
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Logging initialized using configuration in jar:file:/opt/hive/lib/hive-common-2.3.2.jar!/hive-log4j2.properties Async: true
OK
Time taken: 6.456 seconds
Loading data to table mydb.mytab partition (data_date=2019)
OK
Time taken: 1.912 seconds
root@0d2b0044b4c1:/opt# hive -e "use mydb; select * from mytab;"
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Logging initialized using configuration in jar:file:/opt/hive/lib/hive-common-2.3.2.jar!/hive-log4j2.properties Async: true
OK
Time taken: 6.843 seconds
OK
123 ["a","b"] {"1":"a","2":"b"} 2019
root@0d2b0044b4c1:/opt#
這正是我所需要的
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.