Track success or failure for hive queries running through shell script

Question

I am running few hive queries in a single shell script like below

#!/bin/bash  
HQLSource='/home/hql/'  
hive  -f $HQLSource/query1.hql               
hive  -f $HQLSource/query2.hql               

query1.hql and query2.hql have statements to create a table and insert data - if anything fails, I would consider the whole .hql as failed. Can I write to a text file whether the queries are all successful or failed?

Expected Output:

query1.hql - Success  
query2.hql - Failed

Answer 1

The exit code rom the previous command is stored in the variable $? . Any non-zero value is considered a failure of some kind.

#!/bin/bash  
HQLSource='/home/hql/'  
hive -f "$HQLSource/query1.hql"               
query1result=$?
hive -f "$HQLSource/query2.hql"               
query2result=$?
# do stuff with $query1result and $query2result

The exit code is also used by if statements to determine a code path, so you could also do this:

#!/bin/bash  
HQLSource='/home/hql/'  
if hive -f "$HQLSource/query1.hql"
then
    echo Query 1 succeeded
else
    echo Query 1 failed
fi
if hive -f "$HQLSource/query2.hql"
then
    echo Query 2 succeeded
else
    echo Query 2 failed
fi

Answer 2

We can do two checks with in bash file to know status of hql execution.

1. Return result should be equal to 0 else declare hql excution fail.

    beeline -u jdbc:hive2://IP:PORT -n USER -p PASSWORD -f hql_file.hql 2> ./hql_file.logs query1result=$? if [ $query1result -eq 0 ]; then # no issue fi 

2. Scan generated log file for pattern Error/Exception, if pattern matches declare hql fail.

    for logfile in ./hql_file.logs; do errorFound="false" while read -r line; do # check for exception or error.Add grep search pattern based on need. excep=$(echo $line|grep "Error/|Exception") # if no space means no error or exception if [ -n "${excep##+([[:space:]])}" ]; then errorFound="true"; fi done < "$logfile" done 

Answer 3

I have checked and verified the code so use it without any doubt.

1. Put each hive statement in a separate file eg setup_step_4.hive
2. In the shell script call this file. setup_step_4.hive load data local inpath 'nyse.csv' into table nyse_data;

code in shell script will look like this :

hive -f setup_step_3.hive
query1result=$?
echo $query1result
if [ "$query1result" -ne 0 ]
    then
      echo "  Error."
      exit 4
      else 
      echo "Done.Successfully"
    fi

hive -f setup_step_4.hive
query1result=$?
echo $query1result
if [ "$query1result" -ne 0 ]
    then
      echo "  Error"
      exit 4
      else 
      echo "Done.Successfully"
    fi

For example the step 3 executed successfully and the step 4 failed because file does not exists.

Output

Logging initialized using configuration in file:/etc/hive/conf.dist/hive-log4j.properties OK

Time taken: 1.413 seconds

0

Done.Successfully

Logging initialized using configuration in file:/etc/hive/conf.dist/hive-log4j.properties OK

Time taken: 1.574 seconds

FAILED: SemanticException Line 3:23 Invalid path ''nyse.csv'': No files matching path file:/root/Data/nyse.csv

64

Error,