Error using sstableloader on cassandra

Question

I am a cassandra newbie. I see the following error with Cassandra cqlsh 5.0.1 | Cassandra 2.1.2 | CQL spec 3.2.0

Here is the error I see when using sstableloader:

./sstableloader -d <hostname> -u <user> -pw <pass> <filename>
Could not retrieve endpoint ranges: 
java.lang.IllegalArgumentException
java.lang.RuntimeException: Could not retrieve endpoint ranges: 
    at org.apache.cassandra.tools.BulkLoader$ExternalClient.init(BulkLoader.java:337)
    at org.apache.cassandra.io.sstable.SSTableLoader.stream(SSTableLoader.java:157)
    at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:105)
Caused by: java.lang.IllegalArgumentException
    at java.nio.Buffer.limit(Buffer.java:275)
    at org.apache.cassandra.utils.ByteBufferUtil.readBytes(ByteBufferUtil.java:543)
    at org.apache.cassandra.serializers.CollectionSerializer.readValue(CollectionSerializer.java:122)
    at org.apache.cassandra.serializers.MapSerializer.deserializeForNativeProtocol(MapSerializer.java:99)
    at org.apache.cassandra.serializers.MapSerializer.deserializeForNativeProtocol(MapSerializer.java:28)
    at org.apache.cassandra.serializers.CollectionSerializer.deserialize(CollectionSerializer.java:48)
    at org.apache.cassandra.db.marshal.AbstractType.compose(AbstractType.java:66)
    at org.apache.cassandra.cql3.UntypedResultSet$Row.getMap(UntypedResultSet.java:282)
    at org.apache.cassandra.config.CFMetaData.fromSchemaNoTriggers(CFMetaData.java:1793)
    at org.apache.cassandra.config.CFMetaData.fromThriftCqlRow(CFMetaData.java:1101)
    at org.apache.cassandra.tools.BulkLoader$ExternalClient.init(BulkLoader.java:329)
    ... 2 more

What is weird is that I get this error only for a particular keyspace. When I creating a new keyspace (with the same exact command as the issue keyspace and try sstableloader I am not seeing the same issue. When I set DEBUG log level I see the following:

DEBUG [Thrift:1] 2015-02-20 00:32:38,006 CustomTThreadPoolServer.java:212 - Thrift transport error occurred during processing of message.
org.apache.thrift.transport.TTransportException: null
    at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[libthrift-0.9.1.jar:0.9.1]
    at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) ~[libthrift-0.9.1.jar:0.9.1]
    at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129) ~[libthrift-0.9.1.jar:0.9.1]
    at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101) ~[libthrift-0.9.1.jar:0.9.1]
    at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) ~[libthrift-0.9.1.jar:0.9.1]
    at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:362) ~[libthrift-0.9.1.jar:0.9.1]
    at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:284) ~[libthrift-0.9.1.jar:0.9.1]
    at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:191) ~[libthrift-0.9.1.jar:0.9.1]
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27) ~[libthrift-0.9.1.jar:0.9.1]
    at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:202) ~[apache-cassandra-2.1.2.jar:2.1.2]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_65]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_65]
    at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]

Not sure if this is actually an error since per some links online I see that this message appears regardless when setting debug log level

Answer 1

I'm trying this with Cassandra 2.1.8. The error you're seeing is the product of this block of Cassandra code:

try
{
    // Query endpoint to ranges map and schemas from thrift
    InetAddress host = hostiter.next();
    Cassandra.Client client = createThriftClient(host.getHostAddress(), rpcPort, this.user, this.passwd, this.transportFactory);

    setPartitioner(client.describe_partitioner());
    Token.TokenFactory tkFactory = getPartitioner().getTokenFactory();

    for (TokenRange tr : client.describe_ring(keyspace))
    {
        Range<Token> range = new Range<>(tkFactory.fromString(tr.start_token), tkFactory.fromString(tr.end_token), getPartitioner());
        for (String ep : tr.endpoints)
        {
            addRangeForEndpoint(range, InetAddress.getByName(ep));
        }
    }

    String cfQuery = String.format("SELECT * FROM %s.%s WHERE keyspace_name = '%s'",
                                 Keyspace.SYSTEM_KS,
                                 SystemKeyspace.SCHEMA_COLUMNFAMILIES_CF,
                                 keyspace);
    CqlResult cfRes = client.execute_cql3_query(ByteBufferUtil.bytes(cfQuery), Compression.NONE, ConsistencyLevel.ONE);


    for (CqlRow row : cfRes.rows)
    {
        String columnFamily = UTF8Type.instance.getString(row.columns.get(1).bufferForName());
        String columnsQuery = String.format("SELECT * FROM %s.%s WHERE keyspace_name = '%s' AND columnfamily_name = '%s'",
                                            Keyspace.SYSTEM_KS,
                                            SystemKeyspace.SCHEMA_COLUMNS_CF,
                                            keyspace,
                                            columnFamily);
        CqlResult columnsRes = client.execute_cql3_query(ByteBufferUtil.bytes(columnsQuery), Compression.NONE, ConsistencyLevel.ONE);

        CFMetaData metadata = CFMetaData.fromThriftCqlRow(row, columnsRes);
        knownCfs.put(metadata.cfName, metadata);
    }
    break;
}
catch (Exception e)
{
    if (!hostiter.hasNext())
        throw new RuntimeException("Could not retrieve endpoint ranges: ", e);
}

So, what you have is a large variety of errors all rolled up in the message, "Could not retrieve endpoint ranges." You will not be able to tell what your specific error is without downloading the Cassandra source and debugging through it. That's what I did.

My schema is built in a multi-step process using https://github.com/DonBranson/cql_schema_versioning . One step does this:

ALTER TABLE user_reputation DROP ban_votes;

The DROP triggers a Cassandra BulkLoader bug that prints the error message you're seeing. However, the myriad other error conditions show the same message. The error message gives us absolutely nothing to help actually solve the problem.

I also found that the BulkLoader will not work if you're encrypting internode communication like this:

internode_encryption: all

So, in the DigitalOcean cloud where I'm running, where I have to encrypt internode comm, it will fail, but at least it will display a message that indicates a connection failure:

../apache-cassandra-2.1.8/bin/sstableloader -d 192.168.56.101 makeyourcase/arenas
Established connection to initial hosts
Opening sstables and calculating sections to stream
Skipping file makeyourcase-arenas.arenas_name_idx-jb-2-Data.db: column family makeyourcase.arenas.arenas_name_idx doesn't exist
Skipping file makeyourcase-arenas.arenas_name_idx-jb-1-Data.db: column family makeyourcase.arenas.arenas_name_idx doesn't exist
Streaming relevant part of makeyourcase/arenas/makeyourcase-arenas-jb-2-Data.db makeyourcase/arenas/makeyourcase-arenas-jb-1-Data.db to [/192.168.56.102, /192.168.56.101]
ERROR 01:46:39 [Stream #a7e5fb80-3593-11e5-9b52-cdde6a46fde5] Streaming error occurred
java.net.ConnectException: Connection refused
    at sun.nio.ch.Net.connect0(Native Method) ~[na:1.7.0_71]