使用本机协议的Cassandra的自定义map-reduce输入格式化程序

Question

I am using Apache Cassandra (1.2) and Apache Map-Reduce to crunch some data. At the moment I use CqlPagingInputFormat from org.apache.cassandra.hadoop.cql3 . This provider uses Thrift to pull data. It seems that Thrift is fairly slow (300M records, in a 3 node cluster takes 8+ hours to read), and since a native binary protocol exist, I wonder if anyone has used it.

I am not interested in any other optimization and configuration tweaks - that's a separate issue.

My questions are

1. Is there an implementation of a map-reduce input formatter that directly use Cassandra native protocol?

2. If not, what would be the first steps to write my own, for example using a DataStax driver?

Answer 1

Cassandra 2.0.7 includes native protocol analogs for the CQL Hadoop classes:

org.apache.cassandra.hadoop.cql3.CqlInputFormat org.apache.cassandra.hadoop.cql3.CqlRecordReader org.apache.cassandra.hadoop.cql3.CqlConfigHelper

The WordCount code in examples/hadoop_cql3_word_count has been updated to use these classes.

The JIRA that introduced this is https://issues.apache.org/jira/browse/CASSANDRA-6311