Adding the million records into Redis by Jedis lpush - Connection reset by peer: socket write error

Question

When I add one million(1,000,000) into Redis , it is OK.
I get the error Connection reset by peer: socket write error when I add two million(2,000,000) record;

According to Redis data types list ,

The max length of a list is 2 ³² - 1 elements (4294967295, more than 4 billion of elements per list).

/*Creating the json list*/
Gson gson = new GsonBuilder().create();
List<String> employeeList = new ArrayList<String>();
for (int i = 1; i <= 2000000; i++) {
    Employee employee = new Employee(i + "", "Jhon", "My Country", "redis@gmail.com", "+777 92157325");
    String json = gson.toJson(employee);
    employeeList.add(json);
}

/*add json list to Redis*/
Jedis jedis = pool.getResource();
// employeeList size is = 2,000,000
String[] jsonArray = employeeList.toArray(new String[employeeList.size()]);
jedis.lpush("employee_list_1", jsonArray);

Log

Exception in thread "main" redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketException: Connection reset by peer: socket write error
    at redis.clients.jedis.Protocol.sendCommand(Protocol.java:83)
    at redis.clients.jedis.Protocol.sendCommand(Protocol.java:63)
    at redis.clients.jedis.Connection.sendCommand(Connection.java:84)
    at redis.clients.jedis.BinaryClient.lpush(BinaryClient.java:281)
    at redis.clients.jedis.Client.lpush(Client.java:205)
    at redis.clients.jedis.Jedis.lpush(Jedis.java:869)
    at com.mutu.redis.AddJosn.add(AddJosn.java:29)
    at com.mutu.redis.AddJosn.main(AddJosn.java:48)
Caused by: java.net.SocketException: Connection reset by peer: socket write error
    at java.net.SocketOutputStream.socketWrite0(Native Method)
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
    at redis.clients.util.RedisOutputStream.flushBuffer(RedisOutputStream.java:31)
    at redis.clients.util.RedisOutputStream.write(RedisOutputStream.java:54)
    at redis.clients.util.RedisOutputStream.write(RedisOutputStream.java:44)
    at redis.clients.jedis.Protocol.sendCommand(Protocol.java:79)
    ... 7 more

How to add two or more million record by the single transaction/process?

Answer 1

Connection reset by peer means that Redis disconnects the socket connection underlying Jedis instance. Check your Redis log that which warn/error message is logged.

Btw, since lpush with array is sent to one command, request body should be really huge. I think you're doing this just for testing purpose, but I recommend that you need to use multi - exec and break down your request with huge array into lots of requests. It still guarantee atomicity.

If you don't need to have atomicity, you can use pipeline with breaking down your request.