Overflowing mempool during sending a lot of transaction

Question

During sending a lot of transactions using differenr rest servers (we use 2 or 4 nodes/servers) mempool always overflowing and servers starts to overload, the blocks closed for a very long time, it's depending on the amount of transaction, the more transactions the more time for closing blocks need, it always between from 20sec to 20min. For example when i send 20k txs, mempool is filled up to 20k transaction, then after 20 sec block closed with all transactions. If we send 300k transactions, mempool filled up to +-280k, and during 20 minutes closed 3 blocks with ~123, ~123, ~25 txs, but in this 20 minutes tendermint always errors like "err="pong timeout"", "err=EOF", "used of closed network connection" and so on. Also we use cosmos-sdk we can't turn off autogeneration block. https://github.com/tendermint/tendermint/issues/3905 - it is github issue, check it for more information

I tried change nodes configuration, but not much changed

I don't know what I need to do, but we have troubles with mempool and we haven't got resolve. Guys do u have some idea what can I do with this?

Answer 1

Tendermint employs two routines p2p/conn/connection.go#recvRoutine and p2p/conn/connection.go#sendRoutine to handle incoming messages and send responses, and which are both sequential. Therefore, if one node is processing new txs and calling reactors iteratively, it is blocked before all reactors (pl reference to node/node.go#createSwitch for basic reactors).

Tendermint verifies signatures before adding txes to mempool, which takes about .5ms according to config/config.go#DefaultMempoolConfig , so verifying 20k txes literally takes about 10s not including other operation costs.