不带减速器的HBase读/写，异常

Question

I want to read and write hbase without using any reducer.

I followed the example in "The Apache HBase™ Reference Guide", but there are exceptions.

Here is my code:

public class CreateHbaseIndex { 
static final String SRCTABLENAME="sourceTable";
static final String SRCCOLFAMILY="info";
static final String SRCCOL1="name";
static final String SRCCOL2="email";
static final String SRCCOL3="power";

static final String DSTTABLENAME="dstTable";
static final String DSTCOLNAME="index";
static final String DSTCOL1="key";
public static void main(String[] args) {
    System.out.println("CreateHbaseIndex Program starts!...");
    try {
        Configuration config = HBaseConfiguration.create();
        Scan scan = new Scan();
        scan.setCaching(500);
        scan.setCacheBlocks(false);
        scan.addColumn(Bytes.toBytes(SRCCOLFAMILY), Bytes.toBytes(SRCCOL1));//info:name
        HBaseAdmin admin = new HBaseAdmin(config);
        if (admin.tableExists(DSTTABLENAME)) {
            System.out.println("table Exists.");
        }
        else{
            HTableDescriptor tableDesc = new HTableDescriptor(DSTTABLENAME);
            tableDesc.addFamily(new HColumnDescriptor(DSTCOLNAME));
            admin.createTable(tableDesc);
            System.out.println("create table ok.");
        }
        Job job = new Job(config, "CreateHbaseIndex");
        job.setJarByClass(CreateHbaseIndex.class);
        TableMapReduceUtil.initTableMapperJob(
                SRCTABLENAME, // input HBase table name
                scan, // Scan instance to control CF and attribute selection
                HbaseMapper.class, // mapper
                ImmutableBytesWritable.class, // mapper output key
                Put.class, // mapper output value
                job);
        job.waitForCompletion(true);
    } catch (IOException e) {
        e.printStackTrace();
    } catch (InterruptedException e) {
        e.printStackTrace();
    } catch (ClassNotFoundException e) {
        e.printStackTrace();
    }
    System.out.println("Program ends!...");
}

public static class HbaseMapper extends TableMapper<ImmutableBytesWritable, Put> {
    private HTable dstHt;
    private Configuration dstConfig;
    @Override
    public void setup(Context context) throws IOException{
        dstConfig=HBaseConfiguration.create();
        dstHt = new HTable(dstConfig,SRCTABLENAME);
    }

    @Override
    public void map(ImmutableBytesWritable row, Result value, Context context) throws IOException, InterruptedException {
        // this is just copying the data from the source table...
        context.write(row, resultToPut(row,value));
    }

    private static Put resultToPut(ImmutableBytesWritable key, Result result) throws IOException {
        Put put = new Put(key.get());
        for (KeyValue kv : result.raw()) {
            put.add(kv);
        }
        return put;
    }

    @Override
    protected void cleanup(Context context) throws IOException, InterruptedException {
        dstHt.close();
        super.cleanup(context);
    }
}
}

By the way, "souceTable" is like this:

key  name    email
1    peter   a@a.com
2    sam     b@b.com

"dstTable" will be like this:

key    value
peter  1
sam    2

I am a newbie in this field and need you help. Thx~

Answer 1

You are correct that you don't need a reducer to write to HBase, but there are some instances where a reducer might help. If you are creating an index, you might run into situations where two mappers are trying to write the same row. Unless you are careful to ensure that they are writing into different column qualifiers, you could overwrite one update with another due to race conditions. While HBase does do row level locking, it won't help if your application logic is faulty.

Without seeing your exceptions, I would guess that you are failing because you are trying to write key-value pairs from your source table into your index table, where the column family doesn't exist.

Answer 2

In this code you are not specifying the output format. You need to add the following code

    job.setOutputFormatClass(TableOutputFormat.class);

    job.getConfiguration().set(TableOutputFormat.OUTPUT_TABLE,
            DSTTABLENAME);

Also, we are not supposed to create new configuration in the set up, we need to use the same configuration from context.