将流数据插入配置单元

Question

My scenerio is In my project we get 200 msg from kafka every mins and needs to store in hive table using avro format.

If we use insert scriptf or each message, i believe it takes long time to insert for each message. Is there is any idea, for this streaming data which way we have to use to insert into hive with good perforance.

Appreciate your help.

Answer 1

Are you interested in using Apache storm as a streaming mechanism.This can be accomplished using kafka-to-hive bolt using storm which you can easily search for. Please let me know if you are then I can point you in the right direction.

Answer 2

为什么不创建一个外部表并将msg发送到外部表位置。