How to improve the write speed of mongo?

Question

I use mongo and sys-ng to save my log

the log is about 20000 lines pre second

I run mongo whit 2 shard and 3 host with mongos

I write a python script in ipython

In [48]: %time dddd=[qlogdb.insert({'tet':1313232,'test':1232423,'asdasds':'sdadsds'})  for i in range(100000)]
CPU times: user 7.24 s, sys: 2.51 s, total: 9.75 s
Wall time: 10.77 s  

I think it can write 10000 line pre second from the output log

I Need to improve the write speed when insert data to mongo

I want to try use Multiple processes when i insert data to mongo

but i'm not sure if it can improve the write speed ；

any way ?

Answer 1

You require 100,000 write/second and you don't have any requirements on reading the data, ie

• You don't need the data to be consistent by a certain time threshold.
• You don't have any reliability requirements, ie you don't care if you lose the data because N servers crash, and you don't need the writer to know whether the writes were successful.

That being said, there are two pieces of advice I have:

1. Alleviate the I/O bottleneck through the use of RAID , eg RAID 0. This assumes your load is I/O bound; however, your benchmark is also slightly CPU intensive because of the work required to create all the dictionaries.

2. Use batch inserts:

    In [1]: %time return_value = [collection.insert({'tet':1313232,'test':1232423,'asdasds':'sdadsds'}) for i in xrange(100000)] CPU times: user 7.00 s, sys: 0.85 s, total: 7.85 s Wall time: 7.86 s In [2]: %time return_value = [collection.insert({'tet':1313232,'test':1232423,'asdasds':'sdadsds'}) for i in xrange(100000)] CPU times: user 6.79 s, sys: 0.84 s, total: 7.63 s Wall time: 7.64 s In [3]: documents = [{'tet':1313232,'test':1232423,'asdasds':'sdadsds'} for i in xrange(100000)] In [10]: %time return_value = collection.insert(documents) CPU times: user 0.97 s, sys: 0.03 s, total: 0.99 s Wall time: 1.00 s In [4]: documents = [{'tet':1313232,'test':1232423,'asdasds':'sdadsds'} for i in xrange(100000)] In [10]: %time return_value = collection.insert(documents) CPU times: user 0.92 s, sys: 0.04 s, total: 0.96 s Wall time: 0.98 s 

Please note that I replaced your range call with xrange .

Of course if you have read requirements as well then you need to consult the pymongo collection.insert docs with respect to safe and w parameters. No such thing as a free lunch etc.

Hope this helps!