[英]MemoryError in Redis while Pipelining
I am using this python script to migrate data from one ElastiCache redis instance to another. 我正在使用此python脚本将数据从一个ElastiCache redis实例迁移到另一个。 It uses the redis pipelining to migrate data in chunks. 它使用redis流水线方式大块迁移数据。
https://gist.github.com/thomasst/afeda8fe80534a832607 https://gist.github.com/thomasst/afeda8fe80534a832607
But I am getting this strange error: 但是我收到这个奇怪的错误:
Traceback (most recent call last):########### | ETA: 0:00:12
File "migrate-redis.py", line 95, in <module>
migrate()
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 664, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 644, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 837, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 464, in invoke
return callback(*args, **kwargs)
File "migrate-redis.py", line 74, in migrate
results = pipeline.execute(False)
File "/usr/local/lib/python2.7/dist-packages/redis/client.py", line 2593, in execute
return execute(conn, stack, raise_on_error)
File "/usr/local/lib/python2.7/dist-packages/redis/client.py", line 2446, in _execute_transaction
all_cmds = connection.pack_commands([args for args, _ in cmds])
File "/usr/local/lib/python2.7/dist-packages/redis/connection.py", line 637, in pack_commands
output.append(SYM_EMPTY.join(pieces))
MemoryError
There are no issues with RAM as node has 6 GB of RAM. RAM没有问题,因为节点具有6 GB的RAM。
The Memory Profile of source redis is as follows: 源redis的内存配置文件如下:
used_memory:1483900120
used_memory_human:1.38G
used_memory_rss:1945829376
used_memory_peak:2431795528
used_memory_peak_human:2.26G
used_memory_lua:86016
mem_fragmentation_ratio:1.31
mem_allocator:jemalloc-3.6.0
What can be the possible cause for this ? 这可能是什么原因?
From your error log, It has no relation with your redis server. 从错误日志中,它与Redis服务器没有关系。 The error happens in your redis client when it pack all commands into a memory buffer. 当Redis客户端将所有命令打包到内存缓冲区中时,就会发生该错误。
Maybe you could try to decrease the SCAN count option in your migrate-redis.py to test if it is too large to pack it. 也许您可以尝试减少migration-redis.py中的SCAN count选项,以测试它是否太大而无法打包。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.