I am following an tutorial on streaming data from Amazon Kinesis to Hive on Amazon EMR. I was successfully able to do this with a Hive Internal table, but when I try the same with external table I run into a weird EOF error. I have not used STORED BY and STORED AS together before, and I think this could be the issue. I used the IBM guide ( http://www-01.ibm.com/support/knowledgecenter/SSPT3X_2.1.1/com.ibm.swg.im.infosphere.biginsights.bigsql.doc/doc/bsql_create_table.html ) to verify that they can be used together.
Here is the code I am using to build the hive external table.
DROP TABLE apachelog;
CREATE EXTERNAL TABLE apachelog (
host STRING,
IDENTITY STRING,
USER STRING,
TIME STRING,
request STRING,
STATUS STRING,
SIZE STRING,
referrer STRING,
agent STRING
)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe'
WITH SERDEPROPERTIES (
"input.regex" = "([^ ]*) ([^ ]*) ([^ ]*) (-|\\[[^\\]]*\\]) ([^ \"]*|\"[^\"]*\") ([0-9]*) ([0-9]*) ([^ \"]*|\"[^\"]*\") ([^ \"]*|\"[^\"]*\")"
)
STORED AS TEXTFILE
STORED BY
'com.amazon.emr.kinesis.hive.KinesisStorageHandler'
TBLPROPERTIES("kinesis.stream.name"="AccessLogStream")
LOCATION 's3://<bucket-name>/AccessLogStreamOP/';
And here is the error:
hive> source create_access_log.hql;
OK
Time taken: 0.464 seconds
FAILED: ParseException line 19:0 missing EOF at 'STORED' near 'TEXTFILE'
In accordance with Hive LangaugeManual DDL
[
[ROW FORMAT row_format] [STORED AS file_format]
| STORED BY 'storage.handler.class.name' [WITH SERDEPROPERTIES (...)]
]
you may use either ROW FORMAT or STORED BY clause and WITH SERDEPROPERTIES should follow STORED BY
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.