[英]Tokyo cabinet - Slower inserts after hitting 1million
I am evaluating tokyo cabinet Table engine. 我正在评估东京橱柜表引擎。 The insert rate slows down considerable after hitting 1 million records. 在达到100万条记录后,插入率大幅下降。 Batch size is 100,000 and is done within transaction. 批量大小为100,000,在交易中完成。 I tried setting the xmsiz but still no use. 我尝试设置xmsiz但仍然没有用。 Has any one faced this problem with tokyo cabinet? 东京内阁有没有遇到过这个问题?
Details 细节
Tokyo cabinet - 1.4.3 东京内阁 - 1.4.3
Perl bindings - 1.23 Perl绑定 - 1.23
OS : Ubuntu 7.10 (VMWare Player on top of Windows XP) 操作系统:Ubuntu 7.10(Windows XP上的VMWare Player)
I hit a brick wall around 1 million records per shard as well (sharding on the client side, nothing fancy). 我在每个碎片上打了大约100万条记录的砖墙(在客户端进行分片,没什么特别的)。 I tried various ttserver options and they seemed to make no difference, so I looked at the kernel side and found that 我尝试了各种ttserver选项,它们似乎没什么区别,所以我查看了内核方面并发现了
echo 80 > /proc/sys/vm/dirty_ratio
(previous value was 10) gave a big improvement - the following is the total size of the data (on 8 shards, each on its own node) printed every minute: (之前的值为10)给出了很大的改进 - 以下是每分钟打印的数据的总大小(每个节点上有8个分片):
total: 14238792 records, 27.5881 GB size total: 14263546 records, 27.6415 GB size total: 14288997 records, 27.6824 GB size total: 14309739 records, 27.7144 GB size total: 14323563 records, 27.7438 GB size (here I changed the dirty_ratio setting for all shards) total: 14394007 records, 27.8996 GB size total: 14486489 records, 28.0758 GB size total: 14571409 records, 28.2898 GB size total: 14663636 records, 28.4929 GB size total: 14802109 records, 28.7366 GB size
So you can see that the improvement was in the order of 7-8 times. 所以你可以看到改进大约是7-8倍。 Database size was around 4.5GB per node at that point (including indexes) and the nodes have 8GB RAM (so dirty_ratio of 10 meant that the kernel tried to keep less than ca. 800MB dirty). 此时数据库大小约为每个节点4.5GB(包括索引),节点有8GB RAM(因此dirty_ratio为10意味着内核试图保持小于800MB的脏空)。
Next thing I'll try is ext2 (currently: ext3) and noatime and also keeping everything on a ramdisk (that would probably waste twice the amount of memory, but might be worth it). 接下来我要尝试的是ext2(目前:ext3)和noatime并且还将所有内容保存在ramdisk上(这可能会浪费两倍的内存量,但可能值得)。
我只是设置了缓存选项,现在它显着更快。
我认为修改dbtune函数中的bnum参数也会显着提高速度。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.