This is my first question in this forum. I am writing hive batch job logs into a hive log table as-soon-as each step completed. I am using INSERT INTO TABLE
for writing logs into hive table. In hive, multiple records are created for each batch job ID, so I am creating a View to combine logging data collected before using in monitoring tool. Can you please suggest any better solution to achieve this?
Notes:
Batch Job -> Logs -> Hive -> Monitoring
Here is one of the reference architecture I can suggest. You can still use Hive for logging, but use SERDEPROPERTIES
to integrate with HBase
.
Benefits:
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.