Hive query log - aggregated logs per query

Question

Where does Hive store the query logs? There is a property hive.querylog.location from theGetting Started page . This seems to suggest each hive node/client will store the logs in this location. Is there a location where all of these aggregated per query?

Answer 1

By default, Hive stores its logs in /tmp/ currentuser location. When the Hive shell is invoked using the hive command at the Unix/Linux prompt, Hive displays the location of the log using the line Hive history file= location of the log before the Hive prompt is displayed. Example output is shown below:

[root@localhost root]# hive
Logging initialized using configuration in jar:file:/usr/lib/hive/lib/hive-common-0.10.0-cdh4.4.0.jar!/hive-log4j.properties
Hive history file=/tmp/root/hive_job_log_46a27ee2-7ac1-43f5-a66b-20d6ea982421_849420221.txt
hive>

If a Hive query generates MapReduce jobs, those logs can be viewed through the JobTracker page which is accessible via the browser.

The path to the logs is:
Mapper logs: Job Tracker Page -> Completed Jobs -> Jobid -> map -> task id -> Task Logs
Reducer logs: Job Tracker Page -> Completed Jobs -> Jobid -> reduce -> task id -> Task Logs

Answer 2

Execute the query with below command

hive --hiveconf hive.root.logger=DRFA --hiveconf hive.log.dir=./logs --hiveconf hive.log.level=DEBUG -e "<query>"

OR

hive --hiveconf hive.root.logger=DRFA --hiveconf hive.log.dir=./logs --hiveconf hive.log.level=DEBUG -f <hql-file>

It will create a log file in logs folder. Make sure that the logs folder exist in current directory.