How to do Mass insertion in Redis using JAVA?

Question

Hi I need to do multiple insertions of the form

SADD key value

I have the key value pair and needed to know how to perform mass insertions using JAVA. I have written a file in the Redis Protocol. How to proceed further

Answer 1

If you have inputs written to Redis protocol format then why don't just use pipe mode of redis-cli or nc? It's explained from http://redis.io/topics/mass-insert .

If you have mass (key, value) inputs then you can use Jedis to perform sadd with pipelining to get higher performance.

Below example assumes that iter (Iterator) has elements each item is key"\\t"value form.

try (Jedis jedis = new Jedis(host, port)) {
  Pipeline pipeline = jedis.pipelined();
  while (iter.hasNext()) {
    String[] keyValue = iter.next().split("\t");
    pipeline.sadd(keyValue[0], keyValue[1]);
    // you can call pipeline.sync() and start new pipeline here if you think there're so much operations in one pipeline
  }
  pipeline.sync();
}

Answer 2

If you are doing the actual read/write operations through Spring CacheManager with RedisTemplate configured to use Redis as the cache, you can also use the executePipelined method of RedisTemplate which takes a callback as an argument. The callback needs to define the doInRedis method which does the work (read/write operations) in Redis that you want to do in a batch.

Following code shows inserting a List of objects wrapped in a CacheableObject interface that has a getKey() and getValue() by calling redisTemplate.opsForHash().put() .

@Component
public class RedisClient {
  @Autowired
  RedisTemplate redisTemplate;  
  
  //batch-insert using Redis pipeline, a list of objects into the cache specified by cacheName
  public void put(String cacheName, List<CacheableObject> objects) {
    try {
      this.redisTemplate.executePipelined(new RedisCallback<Object>() {
        @Override
        public Object doInRedis(RedisConnection connection) throws DataAccessException {
          for(CacheableObject object: objects) {
            redisTemplate.opsForHash().put(cacheName, object.getKey(), object.getValue()); 
          }
          return null;
        }
      });
    }
    catch(Exception e) {
        log.error("Error inserting objects into Redis cache: {}", e.getMessage());
    }
}

RedisTemplate itself is configured using a configuration class such as the following:

@Configuration
@EnableCaching
public class RedisCacheConfig extends CachingConfigurerSupport implements 
                                                               CachingConfigurer {

  @Value("${redis.hostname}")
  private String redisHost;
  @Value("${redis.port}")
  private int redisPort;
  @Value("${redis.timeout.secs:1}")
  private int redisTimeoutInSecs;
  @Value("${redis.socket.timeout.secs:1}")
  private int redisSocketTimeoutInSecs;
  @Value("${redis.ttl.hours:1}")
  private int redisDataTTL;

  @Bean
  JedisConnectionFactory jedisConnectionFactory() {
     RedisStandaloneConfiguration redisStandaloneConfiguration = new RedisStandaloneConfiguration(redisHost, redisPort);
     return new JedisConnectionFactory(redisStandaloneConfiguration);
  }
 
  @Bean
  public RedisTemplate<Object, Object> redisTemplate() {
    RedisTemplate<Object, Object> redisTemplate = new RedisTemplate<Object, Object>();
    redisTemplate.setConnectionFactory(jedisConnectionFactory());
    return redisTemplate;
}   

  @Bean
  public RedisCacheManager redisCacheManager (JedisConnectionFactory jedisConnectionFactory) {
    RedisCacheConfiguration redisCacheConfiguration = 
            RedisCacheConfiguration.defaultCacheConfig().disableCachingNullValues()
            .entryTtl(Duration.ofHours(redisDataTTL)) .serializeValuesWith(RedisSerializationContext.SerializationPair.fromSerializer(RedisSerializer.java()));
    redisCacheConfiguration.usePrefix();
    RedisCacheManager redisCacheManager = 
            RedisCacheManager.RedisCacheManagerBuilder.
            fromConnectionFactory(jedisConnectionFactory)
            .cacheDefaults(redisCacheConfiguration).build();
    redisCacheManager.setTransactionAware(true);
    return redisCacheManager;
}

  @Bean
  public JedisPoolConfig poolConfig() {
    final JedisPoolConfig jedisPoolConfig = new JedisPoolConfig();
    jedisPoolConfig.setTestOnBorrow(true);
    jedisPoolConfig.setMaxTotal(100);
    jedisPoolConfig.setMaxIdle(100);
    jedisPoolConfig.setMinIdle(10);
    jedisPoolConfig.setTestOnReturn(true);
    jedisPoolConfig.setTestWhileIdle(true);
    return jedisPoolConfig;
}

  @Override
  public CacheErrorHandler errorHandler() {
    return new RedisCacheErrorHandler();
  }
}

Answer 3

Thanks for the clear explanation, I have one doubt here canu please clarify, does it create only one connection or multiple connections with executepipeline? And if i want to just use below for for loop without executepipeline then how it handles the connections.