Hive Protocol Bufferer - NullPointerException while creating table in Hive

Question

Thanks in advance.

Currently we are trying to create Hive table by using Protocol Buffers byte data. We have followed all possible steps of creating Hive table by using Protocol buffer, however getting NullPointerException while creating Hive table. Below are the all required details.

Versions -
1. protoc 3.0.0
2. elephant-bird - 4.14
3. Hortornworks Sandbox Hive version - 1.2.1
4. Protobuf-java 3.0.0

The proto file used is

package tutorial; 
option java_package = "com.mycom.hive.protobuf.serialized";
option java_outer_classname = "BankProtoTest";
message BankClass{
    required string bankAmount= 1;
    required string bankLocation= 2;
    optional string bankName= 3;
}
message BankInfo {
   repeated BankClass bankClass = 1;
}

We are creating the Java class by using below command

protoc.exe -I=input-proto --java_out=java-output input-proto\BankProto.proto

The above command generates the Java class for input protocol buffer file.

After this we copied this protocol buffer java file to a maven java project and then created JAR file. We copied the JAR file into hive lib path ie '/usr/hdp/current/hive-client/lib'.

Below is the create table command

create external table bankproto 
   row format serde "com.twitter.elephantbird.hive.serde.ProtobufDeserializer"     
with serdeproperties 
   ("serialization.class"="com.mycom.hive.protobuf.serialized.BankProtoTest$BankInfo")
stored as  
   inputformat "org.apache.hadoop.mapred.SequenceFileInputFormat" 
   outputformat "org.apache.hadoop.mapred.SequenceFileOutputFormat" 
location '/user/root/protobuf-input/';

The input file present in the location is stored as sequence file in HDFS.
After executing this command getting below exception.

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: MetaException(message:org.apache.hadoop.hive.serde2.SerDeException java.lang.NullPointerException)

Any help related to this is appreciated.

Thanks again.

Avinash Deshmukh

Answer 1

We are able to resolve this issue. The issue was about the correct compatible version of protobuf version. We found that current version of elephant-bird (4.14) is depend on protobuf version 2.6.0