RCFile-发出GZip压缩的int列

Question

For some reason, Hive is not recognizing columns emitted as integers, but does recognize columns emitted as strings.

Is there something about Hive or RCFile or GZ that is preventing proper rendering of int?

My Hive DDL looks like:

create external table if not exists db.table (intField int, strField string) stored as rcfile location '/path/to/my/data';

And the relevant portion of my Java looks like:

BytesRefArrayWritable dataWrite = new BytesRefArrayWritable(2);
byte[] byteArray;
BytesRefWritable bytesRefWritable = new BytesRefWritable();                             intWritable.set(myObj.getIntField());
byteArray = WritableUtils.toByteArray(intWritable.get());
bytesRefWritable.set(byteArray, 0, byteArray.length);
dataWrite.set(0, bytesRefWritable);  // sets int field as column 0


bytesRefWritable = new BytesRefWritable();
textWritable.set(myObj.getStrField());
bytesRefWritable.set(textWritable.getBytes(), 0, textWritable.getLength());
dataWrite.set(1, bytesRefWritable);  // sets str field as column 1

The code runs fine, and through logging I can see the various Writables have bytes within them.

Hive can read the external table as well, but the int field shows up as NULL , indicating some error .

SELECT * from db.table;

OK
NULL    my string field
Time taken: 0.647 seconds

Any idea what might be going on here?

Answer 1

So, I'm not sure exactly why this is the case, but I got it working using the following method:

In the code that writes the byte array representing the integer value, instead of using WritableUtils.toByteArray() , I instead Text.set(Integer.toString(intVal)).getBytes() .

In other words, I convert the integer to its String representation, and use the Text writable object to get the byte array as if it were a string.

Then, in my Hive DDL, I can call the column an int and it interprets it correctly.

I'm not sure what was initially causing the problem, be it a bug in WritableUtils , some incompatibility with compressed integer byte arrays, or a faulty understanding of how this stuff works on my part. In any event, the solution described above successfully meets the task's needs.