spring data redis cluster pipeline support

Question

Is there a plan to support "pipelined" operations for the spring data redis library when connecting to redis clustered version. There is considerable performance differences between pipelined & non-pipelined operations. If there is no such plan what are other viable options?

Answer 1

Spring Data Redis provides several RedisTemplate methods for executing commands in a pipeline. If you don't care about the results of the pipelined operations, you can use the standard execute method, passing true for the pipeline argument. The executePipelined methods will execute the provided RedisCallback or SessionCallback in a pipeline and return the results. For example:

//pop a specified number of items from a queue
List<Object> results = stringRedisTemplate.executePipelined(
  new RedisCallback<Object>() {
    public Object doInRedis(RedisConnection connection) throws DataAccessException {
      StringRedisConnection stringRedisConn = (StringRedisConnection)connection;
      for(int i=0; i< batchSize; i++) {
        stringRedisConn.rPop("myqueue");
      }
    return null;
  }
});

The example above executes a bulk right pop of items from a queue in a pipeline. The results List contains all of the popped items. RedisTemplate uses its value, hash key, and hash value serializers to deserialize all results before returning, so the returned items in the above example will be Strings. There are additional executePipelined methods that allow you to pass a custom serializer for pipelined results.

Note that the value returned from the RedisCallback is required to be null, as this value is discarded in favor of returning the results of the pipelined commands.

Reference: http://docs.spring.io/spring-data/redis/docs/current/reference/html/#pipeline

Answer 2

Spring data redis does not supports the pipeline on cluster. So We can do it by our self in our application. If you use spring-data-redis and Jedis library.

So for that, we have to take Jedis Connection indirectly from Jedis Pool. Now if you only know keys of cluster then for that first you need to find out slot associated to key. You can get it by following way.

int slot = JedisClusterCRC16.getSlot(hKey);

Second, you can get JedisCluster connection by following way

JedisClusterConnection jedisClusterConnection = (JedisClusterConnection)stringRedisTemplate.getConnectionFactory().getClusterConnection();
JedisCluster jedisCluster = jedisClusterConnection.getNativeConnection();

Now you have JedisCluster connection but it is not enough to get Jedis connection from Jedis pool. Because JedisCluster not exposing connection handler directly and JedisSlotConnectionHandler classing having method which returns the jedis connection from slot. so for that we have to copy the BinaryJedisCluster class from package redis.clients.jedis into our application with same class and package name and you have to add following method to expose connection handler.

public JedisSlotBasedConnectionHandler getConnectionHandler() {
   return (JedisSlotBasedConnectionHandler) this.connectionHandler;
}

Finally, you can able to get Jedis Connection by calling getJedisConnectionFromSlot(slot) method

JedisSlotBasedConnectionHandler jedisClusterConnectionHandler = jedisCluster.getConnectionHandler();
Jedis connection = jedisClusterConnectionHandler.getConnectionFromSlot(slot);
Pipeline pipeline = connection.pipelined();
pipeline.somecommand....
pipeline.sync();

Reference link