Kyoto Cabinet HashDB inserts are not O(1)

Question

I am experimenting with Kyoto Cabinet HashDB. My database will grow up to the size of 110 TB. The keys are 20 Bytes and values are 128 KB. I am trying to insert about 500 million records. Here is the best performance I have gotten so far:

Time is in seconds. 'n' is the number of 10000 records entered. For example 1st 10000 records took 1 second. 5th 10000 records took 5 seconds and so on. As you can see the performance starts degrading after about 2 million records are entered. I created the database with bucket number set to 7 Billion, and everything else is default. I am using the C/C++ library, and setting memory map capacity to 30 GB (I have 32 GB RAM) using db.tune_map(30LL << 30); .

Inserted 10000 records. n: 1. Time: 1
Inserted 10000 records. n: 2. Time: 1
Inserted 10000 records. n: 3. Time: 1
Inserted 10000 records. n: 4. Time: 1
Inserted 10000 records. n: 5. Time: 4
Inserted 10000 records. n: 6. Time: 4
Inserted 10000 records. n: 7. Time: 4
Inserted 10000 records. n: 8. Time: 4
Inserted 10000 records. n: 9. Time: 4
Inserted 10000 records. n: 10. Time: 4
Inserted 10000 records. n: 11. Time: 4
Inserted 10000 records. n: 12. Time: 4
Inserted 10000 records. n: 13. Time: 4
Inserted 10000 records. n: 14. Time: 4
Inserted 10000 records. n: 15. Time: 4
Inserted 10000 records. n: 16. Time: 4
Inserted 10000 records. n: 17. Time: 4
Inserted 10000 records. n: 18. Time: 4
Inserted 10000 records. n: 19. Time: 7
Inserted 10000 records. n: 20. Time: 4
Inserted 10000 records. n: 21. Time: 4
Inserted 10000 records. n: 22. Time: 4
Inserted 10000 records. n: 23. Time: 5
Inserted 10000 records. n: 24. Time: 4
Inserted 10000 records. n: 25. Time: 4
Inserted 10000 records. n: 26. Time: 4
Inserted 10000 records. n: 27. Time: 4
Inserted 10000 records. n: 28. Time: 4
Inserted 10000 records. n: 29. Time: 5
Inserted 10000 records. n: 30. Time: 4
Inserted 10000 records. n: 31. Time: 4
Inserted 10000 records. n: 32. Time: 4
Inserted 10000 records. n: 33. Time: 4
Inserted 10000 records. n: 34. Time: 4
Inserted 10000 records. n: 35. Time: 4
Inserted 10000 records. n: 36. Time: 4
Inserted 10000 records. n: 37. Time: 5
Inserted 10000 records. n: 38. Time: 4
Inserted 10000 records. n: 39. Time: 4
Inserted 10000 records. n: 40. Time: 4
Inserted 10000 records. n: 41. Time: 4
Inserted 10000 records. n: 42. Time: 4
Inserted 10000 records. n: 43. Time: 4
Inserted 10000 records. n: 44. Time: 4
Inserted 10000 records. n: 45. Time: 4
Inserted 10000 records. n: 46. Time: 4
Inserted 10000 records. n: 47. Time: 4
Inserted 10000 records. n: 48. Time: 5
Inserted 10000 records. n: 49. Time: 4
Inserted 10000 records. n: 50. Time: 4
Inserted 10000 records. n: 51. Time: 5
Inserted 10000 records. n: 52. Time: 4
Inserted 10000 records. n: 53. Time: 5
Inserted 10000 records. n: 54. Time: 5
Inserted 10000 records. n: 55. Time: 4
Inserted 10000 records. n: 56. Time: 5
Inserted 10000 records. n: 57. Time: 5
Inserted 10000 records. n: 58. Time: 5
Inserted 10000 records. n: 59. Time: 5
Inserted 10000 records. n: 60. Time: 5
Inserted 10000 records. n: 61. Time: 4
Inserted 10000 records. n: 62. Time: 5
Inserted 10000 records. n: 63. Time: 5
Inserted 10000 records. n: 64. Time: 4
Inserted 10000 records. n: 65. Time: 6
Inserted 10000 records. n: 66. Time: 4
Inserted 10000 records. n: 67. Time: 4
Inserted 10000 records. n: 68. Time: 7
Inserted 10000 records. n: 69. Time: 6
Inserted 10000 records. n: 70. Time: 7
Inserted 10000 records. n: 71. Time: 7
Inserted 10000 records. n: 72. Time: 6
Inserted 10000 records. n: 73. Time: 10
Inserted 10000 records. n: 74. Time: 7
Inserted 10000 records. n: 75. Time: 10
Inserted 10000 records. n: 76. Time: 9
Inserted 10000 records. n: 77. Time: 8
Inserted 10000 records. n: 78. Time: 10
Inserted 10000 records. n: 79. Time: 8
Inserted 10000 records. n: 80. Time: 6
Inserted 10000 records. n: 81. Time: 8
Inserted 10000 records. n: 82. Time: 8
Inserted 10000 records. n: 83. Time: 9
Inserted 10000 records. n: 84. Time: 9
Inserted 10000 records. n: 85. Time: 9
Inserted 10000 records. n: 86. Time: 9
Inserted 10000 records. n: 87. Time: 7
Inserted 10000 records. n: 88. Time: 10
Inserted 10000 records. n: 89. Time: 8
Inserted 10000 records. n: 90. Time: 10
Inserted 10000 records. n: 91. Time: 10
Inserted 10000 records. n: 92. Time: 10
Inserted 10000 records. n: 93. Time: 10
Inserted 10000 records. n: 94. Time: 10
Inserted 10000 records. n: 95. Time: 10
Inserted 10000 records. n: 96. Time: 11
Inserted 10000 records. n: 97. Time: 9
Inserted 10000 records. n: 98. Time: 10
Inserted 10000 records. n: 99. Time: 10
Inserted 10000 records. n: 100. Time: 11
Inserted 10000 records. n: 101. Time: 10
Inserted 10000 records. n: 102. Time: 11
Inserted 10000 records. n: 103. Time: 11
Inserted 10000 records. n: 104. Time: 11
Inserted 10000 records. n: 105. Time: 12
Inserted 10000 records. n: 106. Time: 11
Inserted 10000 records. n: 107. Time: 12
Inserted 10000 records. n: 108. Time: 12
Inserted 10000 records. n: 109. Time: 10
Inserted 10000 records. n: 110. Time: 11
Inserted 10000 records. n: 111. Time: 10
Inserted 10000 records. n: 112. Time: 10
Inserted 10000 records. n: 113. Time: 11
Inserted 10000 records. n: 114. Time: 10
Inserted 10000 records. n: 115. Time: 11
Inserted 10000 records. n: 116. Time: 10
Inserted 10000 records. n: 117. Time: 10
Inserted 10000 records. n: 118. Time: 10
Inserted 10000 records. n: 119. Time: 10
Inserted 10000 records. n: 120. Time: 11
Inserted 10000 records. n: 121. Time: 10
Inserted 10000 records. n: 122. Time: 11
Inserted 10000 records. n: 123. Time: 9
Inserted 10000 records. n: 124. Time: 11
Inserted 10000 records. n: 125. Time: 9
Inserted 10000 records. n: 126. Time: 11
Inserted 10000 records. n: 127. Time: 10
Inserted 10000 records. n: 128. Time: 10
Inserted 10000 records. n: 129. Time: 12
Inserted 10000 records. n: 130. Time: 10
Inserted 10000 records. n: 131. Time: 10
Inserted 10000 records. n: 132. Time: 10
Inserted 10000 records. n: 133. Time: 10
Inserted 10000 records. n: 134. Time: 16
Inserted 10000 records. n: 135. Time: 10
Inserted 10000 records. n: 136. Time: 10
Inserted 10000 records. n: 137. Time: 10
Inserted 10000 records. n: 138. Time: 10
Inserted 10000 records. n: 139. Time: 9
Inserted 10000 records. n: 140. Time: 10
Inserted 10000 records. n: 141. Time: 10
Inserted 10000 records. n: 142. Time: 10
Inserted 10000 records. n: 143. Time: 10
Inserted 10000 records. n: 144. Time: 11
Inserted 10000 records. n: 145. Time: 10
Inserted 10000 records. n: 146. Time: 11
Inserted 10000 records. n: 147. Time: 9
Inserted 10000 records. n: 148. Time: 10
Inserted 10000 records. n: 149. Time: 9
Inserted 10000 records. n: 150. Time: 10
Inserted 10000 records. n: 151. Time: 20
Inserted 10000 records. n: 152. Time: 18
Inserted 10000 records. n: 153. Time: 33
Inserted 10000 records. n: 154. Time: 25
Inserted 10000 records. n: 155. Time: 30
Inserted 10000 records. n: 156. Time: 18
Inserted 10000 records. n: 157. Time: 12
Inserted 10000 records. n: 158. Time: 13
Inserted 10000 records. n: 159. Time: 14
Inserted 10000 records. n: 160. Time: 10
Inserted 10000 records. n: 161. Time: 9
Inserted 10000 records. n: 162. Time: 12
Inserted 10000 records. n: 163. Time: 11
Inserted 10000 records. n: 164. Time: 11
Inserted 10000 records. n: 165. Time: 11
Inserted 10000 records. n: 166. Time: 11
Inserted 10000 records. n: 167. Time: 12
Inserted 10000 records. n: 168. Time: 10
Inserted 10000 records. n: 169. Time: 11
Inserted 10000 records. n: 170. Time: 12
Inserted 10000 records. n: 171. Time: 11
Inserted 10000 records. n: 172. Time: 10
Inserted 10000 records. n: 173. Time: 11
Inserted 10000 records. n: 174. Time: 11
Inserted 10000 records. n: 175. Time: 10
Inserted 10000 records. n: 176. Time: 11
Inserted 10000 records. n: 177. Time: 11
Inserted 10000 records. n: 178. Time: 10
Inserted 10000 records. n: 179. Time: 11
Inserted 10000 records. n: 180. Time: 11
Inserted 10000 records. n: 181. Time: 11
Inserted 10000 records. n: 182. Time: 10
Inserted 10000 records. n: 183. Time: 11
Inserted 10000 records. n: 184. Time: 11
Inserted 10000 records. n: 185. Time: 11
Inserted 10000 records. n: 186. Time: 11
Inserted 10000 records. n: 187. Time: 11
Inserted 10000 records. n: 188. Time: 11
Inserted 10000 records. n: 189. Time: 11
Inserted 10000 records. n: 190. Time: 11
Inserted 10000 records. n: 191. Time: 11
Inserted 10000 records. n: 192. Time: 13
Inserted 10000 records. n: 193. Time: 13
Inserted 10000 records. n: 194. Time: 13
Inserted 10000 records. n: 195. Time: 12
Inserted 10000 records. n: 196. Time: 13
Inserted 10000 records. n: 197. Time: 11
Inserted 10000 records. n: 198. Time: 11
Inserted 10000 records. n: 199. Time: 12
Inserted 10000 records. n: 200. Time: 11
Inserted 10000 records. n: 201. Time: 11
Inserted 10000 records. n: 202. Time: 11
Inserted 10000 records. n: 203. Time: 11
Inserted 10000 records. n: 204. Time: 11
Inserted 10000 records. n: 205. Time: 12
Inserted 10000 records. n: 206. Time: 11
Inserted 10000 records. n: 207. Time: 10
Inserted 10000 records. n: 208. Time: 10
Inserted 10000 records. n: 209. Time: 11
Inserted 10000 records. n: 210. Time: 10
Inserted 10000 records. n: 211. Time: 10
Inserted 10000 records. n: 212. Time: 11
Inserted 10000 records. n: 213. Time: 11
Inserted 10000 records. n: 214. Time: 10
Inserted 10000 records. n: 215. Time: 11
Inserted 10000 records. n: 216. Time: 11
Inserted 10000 records. n: 217. Time: 12
Inserted 10000 records. n: 218. Time: 12
Inserted 10000 records. n: 219. Time: 12
Inserted 10000 records. n: 220. Time: 11
Inserted 10000 records. n: 221. Time: 12
Inserted 10000 records. n: 222. Time: 10
Inserted 10000 records. n: 223. Time: 11
Inserted 10000 records. n: 224. Time: 11
Inserted 10000 records. n: 225. Time: 11
Inserted 10000 records. n: 226. Time: 11
Inserted 10000 records. n: 227. Time: 11
Inserted 10000 records. n: 228. Time: 11
Inserted 10000 records. n: 229. Time: 11
Inserted 10000 records. n: 230. Time: 11
Inserted 10000 records. n: 231. Time: 10
Inserted 10000 records. n: 232. Time: 11
Inserted 10000 records. n: 233. Time: 10
Inserted 10000 records. n: 234. Time: 11
Inserted 10000 records. n: 235. Time: 11
Inserted 10000 records. n: 236. Time: 11
Inserted 10000 records. n: 237. Time: 11
Inserted 10000 records. n: 238. Time: 10
Inserted 10000 records. n: 239. Time: 11
Inserted 10000 records. n: 240. Time: 11
Inserted 10000 records. n: 241. Time: 11
Inserted 10000 records. n: 242. Time: 11
Inserted 10000 records. n: 243. Time: 11
Inserted 10000 records. n: 244. Time: 11
Inserted 10000 records. n: 245. Time: 11
Inserted 10000 records. n: 246. Time: 12
Inserted 10000 records. n: 247. Time: 11
Inserted 10000 records. n: 248. Time: 12
Inserted 10000 records. n: 249. Time: 12
Inserted 10000 records. n: 250. Time: 11
Inserted 10000 records. n: 251. Time: 12
Inserted 10000 records. n: 252. Time: 12
Inserted 10000 records. n: 253. Time: 13
Inserted 10000 records. n: 254. Time: 10
Inserted 10000 records. n: 255. Time: 12
Inserted 10000 records. n: 256. Time: 12
Inserted 10000 records. n: 257. Time: 13
Inserted 10000 records. n: 258. Time: 11
Inserted 10000 records. n: 259. Time: 10
Inserted 10000 records. n: 260. Time: 12
Inserted 10000 records. n: 261. Time: 11
Inserted 10000 records. n: 262. Time: 11
Inserted 10000 records. n: 263. Time: 12
Inserted 10000 records. n: 264. Time: 10
Inserted 10000 records. n: 265. Time: 10
Inserted 10000 records. n: 266. Time: 11
Inserted 10000 records. n: 267. Time: 10
Inserted 10000 records. n: 268. Time: 11
Inserted 10000 records. n: 269. Time: 12
Inserted 10000 records. n: 270. Time: 12
Inserted 10000 records. n: 271. Time: 11
Inserted 10000 records. n: 272. Time: 11
Inserted 10000 records. n: 273. Time: 10
Inserted 10000 records. n: 274. Time: 12
Inserted 10000 records. n: 275. Time: 9
Inserted 10000 records. n: 276. Time: 12
Inserted 10000 records. n: 277. Time: 11
Inserted 10000 records. n: 278. Time: 11
Inserted 10000 records. n: 279. Time: 11
Inserted 10000 records. n: 280. Time: 10
Inserted 10000 records. n: 281. Time: 11
Inserted 10000 records. n: 282. Time: 11
Inserted 10000 records. n: 283. Time: 10
Inserted 10000 records. n: 284. Time: 10
Inserted 10000 records. n: 285. Time: 11
Inserted 10000 records. n: 286. Time: 10
Inserted 10000 records. n: 287. Time: 10
Inserted 10000 records. n: 288. Time: 11
Inserted 10000 records. n: 289. Time: 10
Inserted 10000 records. n: 290. Time: 9
Inserted 10000 records. n: 291. Time: 16
Inserted 10000 records. n: 292. Time: 10
Inserted 10000 records. n: 293. Time: 9
Inserted 10000 records. n: 294. Time: 11
Inserted 10000 records. n: 295. Time: 9
Inserted 10000 records. n: 296. Time: 9
Inserted 10000 records. n: 297. Time: 11
Inserted 10000 records. n: 298. Time: 39
Inserted 10000 records. n: 299. Time: 52
Inserted 10000 records. n: 300. Time: 54
Inserted 10000 records. n: 301. Time: 42
Inserted 10000 records. n: 302. Time: 42
Inserted 10000 records. n: 303. Time: 11
Inserted 10000 records. n: 304. Time: 9
Inserted 10000 records. n: 305. Time: 7
Inserted 10000 records. n: 306. Time: 8
Inserted 10000 records. n: 307. Time: 10
Inserted 10000 records. n: 308. Time: 10
Inserted 10000 records. n: 309. Time: 10
Inserted 10000 records. n: 310. Time: 10
Inserted 10000 records. n: 311. Time: 9
Inserted 10000 records. n: 312. Time: 11
Inserted 10000 records. n: 313. Time: 11
Inserted 10000 records. n: 314. Time: 13
Inserted 10000 records. n: 315. Time: 13
Inserted 10000 records. n: 316. Time: 12
Inserted 10000 records. n: 317. Time: 11
Inserted 10000 records. n: 318. Time: 10
Inserted 10000 records. n: 319. Time: 9
Inserted 10000 records. n: 320. Time: 12
Inserted 10000 records. n: 321. Time: 11
Inserted 10000 records. n: 322. Time: 12
Inserted 10000 records. n: 323. Time: 10
Inserted 10000 records. n: 324. Time: 11
Inserted 10000 records. n: 325. Time: 11
Inserted 10000 records. n: 326. Time: 13
Inserted 10000 records. n: 327. Time: 13
Inserted 10000 records. n: 328. Time: 12
Inserted 10000 records. n: 329. Time: 12
Inserted 10000 records. n: 330. Time: 10
Inserted 10000 records. n: 331. Time: 11
Inserted 10000 records. n: 332. Time: 10
Inserted 10000 records. n: 333. Time: 11
Inserted 10000 records. n: 334. Time: 11
Inserted 10000 records. n: 335. Time: 11
Inserted 10000 records. n: 336. Time: 10
Inserted 10000 records. n: 337. Time: 11
Inserted 10000 records. n: 338. Time: 12
Inserted 10000 records. n: 339. Time: 13
Inserted 10000 records. n: 340. Time: 14
Inserted 10000 records. n: 341. Time: 13
Inserted 10000 records. n: 342. Time: 10
Inserted 10000 records. n: 343. Time: 10
Inserted 10000 records. n: 344. Time: 9
Inserted 10000 records. n: 345. Time: 11
Inserted 10000 records. n: 346. Time: 12
Inserted 10000 records. n: 347. Time: 12
Inserted 10000 records. n: 348. Time: 10
Inserted 10000 records. n: 349. Time: 11
Inserted 10000 records. n: 350. Time: 13
Inserted 10000 records. n: 351. Time: 13
Inserted 10000 records. n: 352. Time: 11
Inserted 10000 records. n: 353. Time: 12
Inserted 10000 records. n: 354. Time: 10
Inserted 10000 records. n: 355. Time: 10
Inserted 10000 records. n: 356. Time: 12
Inserted 10000 records. n: 357. Time: 11
Inserted 10000 records. n: 358. Time: 11
Inserted 10000 records. n: 359. Time: 12
Inserted 10000 records. n: 360. Time: 11
Inserted 10000 records. n: 361. Time: 12
Inserted 10000 records. n: 362. Time: 13
Inserted 10000 records. n: 363. Time: 12
Inserted 10000 records. n: 364. Time: 11
Inserted 10000 records. n: 365. Time: 11
Inserted 10000 records. n: 366. Time: 11
Inserted 10000 records. n: 367. Time: 11
Inserted 10000 records. n: 368. Time: 12
Inserted 10000 records. n: 369. Time: 11
Inserted 10000 records. n: 370. Time: 14
Inserted 10000 records. n: 371. Time: 14
Inserted 10000 records. n: 372. Time: 15
Inserted 10000 records. n: 373. Time: 16
Inserted 10000 records. n: 374. Time: 18
Inserted 10000 records. n: 375. Time: 15
Inserted 10000 records. n: 376. Time: 11
Inserted 10000 records. n: 377. Time: 11
Inserted 10000 records. n: 378. Time: 11
Inserted 10000 records. n: 379. Time: 12
Inserted 10000 records. n: 380. Time: 11
Inserted 10000 records. n: 381. Time: 12
Inserted 10000 records. n: 382. Time: 12
Inserted 10000 records. n: 383. Time: 14
Inserted 10000 records. n: 384. Time: 12
Inserted 10000 records. n: 385. Time: 16
Inserted 10000 records. n: 386. Time: 14
Inserted 10000 records. n: 387. Time: 10
Inserted 10000 records. n: 388. Time: 12
Inserted 10000 records. n: 389. Time: 11
Inserted 10000 records. n: 390. Time: 11
Inserted 10000 records. n: 391. Time: 11
Inserted 10000 records. n: 392. Time: 12
Inserted 10000 records. n: 393. Time: 15
Inserted 10000 records. n: 394. Time: 16
Inserted 10000 records. n: 395. Time: 17
Inserted 10000 records. n: 396. Time: 13
Inserted 10000 records. n: 397. Time: 13
Inserted 10000 records. n: 398. Time: 17
Inserted 10000 records. n: 399. Time: 11
Inserted 10000 records. n: 400. Time: 16
Inserted 10000 records. n: 401. Time: 12
Inserted 10000 records. n: 402. Time: 13
Inserted 10000 records. n: 403. Time: 13
Inserted 10000 records. n: 404. Time: 13
Inserted 10000 records. n: 405. Time: 16
Inserted 10000 records. n: 406. Time: 10
Inserted 10000 records. n: 407. Time: 12
Inserted 10000 records. n: 408. Time: 14
Inserted 10000 records. n: 409. Time: 12
Inserted 10000 records. n: 410. Time: 18
Inserted 10000 records. n: 411. Time: 13
Inserted 10000 records. n: 412. Time: 20
Inserted 10000 records. n: 413. Time: 14
Inserted 10000 records. n: 414. Time: 15
Inserted 10000 records. n: 415. Time: 52
Inserted 10000 records. n: 416. Time: 22
Inserted 10000 records. n: 417. Time: 111
Inserted 10000 records. n: 418. Time: 81
Inserted 10000 records. n: 419. Time: 80
Inserted 10000 records. n: 420. Time: 78
Inserted 10000 records. n: 421. Time: 116
Inserted 10000 records. n: 422. Time: 123
Inserted 10000 records. n: 423. Time: 86
Inserted 10000 records. n: 424. Time: 24
Inserted 10000 records. n: 425. Time: 36
Inserted 10000 records. n: 426. Time: 21
Inserted 10000 records. n: 427. Time: 15
Inserted 10000 records. n: 428. Time: 17
Inserted 10000 records. n: 429. Time: 15
Inserted 10000 records. n: 430. Time: 15
Inserted 10000 records. n: 431. Time: 16
Inserted 10000 records. n: 432. Time: 15
Inserted 10000 records. n: 433. Time: 18
Inserted 10000 records. n: 434. Time: 20
Inserted 10000 records. n: 435. Time: 18
Inserted 10000 records. n: 436. Time: 20
Inserted 10000 records. n: 437. Time: 20
Inserted 10000 records. n: 438. Time: 22
Inserted 10000 records. n: 439. Time: 22
Inserted 10000 records. n: 440. Time: 26
Inserted 10000 records. n: 441. Time: 24
Inserted 10000 records. n: 442. Time: 69
Inserted 10000 records. n: 443. Time: 133
Inserted 10000 records. n: 444. Time: 17
Inserted 10000 records. n: 445. Time: 18
Inserted 10000 records. n: 446. Time: 20
Inserted 10000 records. n: 447. Time: 19
Inserted 10000 records. n: 448. Time: 19
Inserted 10000 records. n: 449. Time: 20
Inserted 10000 records. n: 450. Time: 18
Inserted 10000 records. n: 451. Time: 18
Inserted 10000 records. n: 452. Time: 19
Inserted 10000 records. n: 453. Time: 19
Inserted 10000 records. n: 454. Time: 19
Inserted 10000 records. n: 455. Time: 22
Inserted 10000 records. n: 456. Time: 33
Inserted 10000 records. n: 457. Time: 25
Inserted 10000 records. n: 458. Time: 23
Inserted 10000 records. n: 459. Time: 24
Inserted 10000 records. n: 460. Time: 134
Inserted 10000 records. n: 461. Time: 31
Inserted 10000 records. n: 462. Time: 21
Inserted 10000 records. n: 463. Time: 21
Inserted 10000 records. n: 464. Time: 21
Inserted 10000 records. n: 465. Time: 20
Inserted 10000 records. n: 466. Time: 25
Inserted 10000 records. n: 467. Time: 26
Inserted 10000 records. n: 468. Time: 24
Inserted 10000 records. n: 469. Time: 25
Inserted 10000 records. n: 470. Time: 31
Inserted 10000 records. n: 471. Time: 28
Inserted 10000 records. n: 472. Time: 31
Inserted 10000 records. n: 473. Time: 32
Inserted 10000 records. n: 474. Time: 33
Inserted 10000 records. n: 475. Time: 31
Inserted 10000 records. n: 476. Time: 40
Inserted 10000 records. n: 477. Time: 120
Inserted 10000 records. n: 478. Time: 23
Inserted 10000 records. n: 479. Time: 30
Inserted 10000 records. n: 480. Time: 33
Inserted 10000 records. n: 481. Time: 28
Inserted 10000 records. n: 482. Time: 26
Inserted 10000 records. n: 483. Time: 31
Inserted 10000 records. n: 484. Time: 22
Inserted 10000 records. n: 485. Time: 26
Inserted 10000 records. n: 486. Time: 25
Inserted 10000 records. n: 487. Time: 22
Inserted 10000 records. n: 488. Time: 38
Inserted 10000 records. n: 489. Time: 24
Inserted 10000 records. n: 490. Time: 27
Inserted 10000 records. n: 491. Time: 42
Inserted 10000 records. n: 492. Time: 39
Inserted 10000 records. n: 493. Time: 97
Inserted 10000 records. n: 494. Time: 59
Inserted 10000 records. n: 495. Time: 28
Inserted 10000 records. n: 496. Time: 24
Inserted 10000 records. n: 497. Time: 25
Inserted 10000 records. n: 498. Time: 24
Inserted 10000 records. n: 499. Time: 37
Inserted 10000 records. n: 500. Time: 24
Inserted 10000 records. n: 501. Time: 30
Inserted 10000 records. n: 502. Time: 31
Inserted 10000 records. n: 503. Time: 38
Inserted 10000 records. n: 504. Time: 65
Inserted 10000 records. n: 505. Time: 61
Inserted 10000 records. n: 506. Time: 47
Inserted 10000 records. n: 507. Time: 31
Inserted 10000 records. n: 508. Time: 38
Inserted 10000 records. n: 509. Time: 29
Inserted 10000 records. n: 510. Time: 31
Inserted 10000 records. n: 511. Time: 30
Inserted 10000 records. n: 512. Time: 33
Inserted 10000 records. n: 513. Time: 30
Inserted 10000 records. n: 514. Time: 35
Inserted 10000 records. n: 515. Time: 35
Inserted 10000 records. n: 516. Time: 33
Inserted 10000 records. n: 517. Time: 33
Inserted 10000 records. n: 518. Time: 30
Inserted 10000 records. n: 519. Time: 46
Inserted 10000 records. n: 520. Time: 75
Inserted 10000 records. n: 521. Time: 30
Inserted 10000 records. n: 522. Time: 36
Inserted 10000 records. n: 523. Time: 28
Inserted 10000 records. n: 524. Time: 29
Inserted 10000 records. n: 525. Time: 32
Inserted 10000 records. n: 526. Time: 34
Inserted 10000 records. n: 527. Time: 43
Inserted 10000 records. n: 528. Time: 38
Inserted 10000 records. n: 529. Time: 37
Inserted 10000 records. n: 530. Time: 29
Inserted 10000 records. n: 531. Time: 29
Inserted 10000 records. n: 532. Time: 30
Inserted 10000 records. n: 533. Time: 33
Inserted 10000 records. n: 534. Time: 39
Inserted 10000 records. n: 535. Time: 55
Inserted 10000 records. n: 536. Time: 73
Inserted 10000 records. n: 537. Time: 50
Inserted 10000 records. n: 538. Time: 54
Inserted 10000 records. n: 539. Time: 31
Inserted 10000 records. n: 540. Time: 38
Inserted 10000 records. n: 541. Time: 50
Inserted 10000 records. n: 542. Time: 40
Inserted 10000 records. n: 543. Time: 21
Inserted 10000 records. n: 544. Time: 38
Inserted 10000 records. n: 545. Time: 34
Inserted 10000 records. n: 546. Time: 41
Inserted 10000 records. n: 547. Time: 34
Inserted 10000 records. n: 548. Time: 31
Inserted 10000 records. n: 549. Time: 37
Inserted 10000 records. n: 550. Time: 58
Inserted 10000 records. n: 551. Time: 42
Inserted 10000 records. n: 552. Time: 36
Inserted 10000 records. n: 553. Time: 38
Inserted 10000 records. n: 554. Time: 55
Inserted 10000 records. n: 555. Time: 39
Inserted 10000 records. n: 556. Time: 45
Inserted 10000 records. n: 557. Time: 36
Inserted 10000 records. n: 558. Time: 48
Inserted 10000 records. n: 559. Time: 45
Inserted 10000 records. n: 560. Time: 36
Inserted 10000 records. n: 561. Time: 32
Inserted 10000 records. n: 562. Time: 36
Inserted 10000 records. n: 563. Time: 46
Inserted 10000 records. n: 564. Time: 51
Inserted 10000 records. n: 565. Time: 49
Inserted 10000 records. n: 566. Time: 47
Inserted 10000 records. n: 567. Time: 49
Inserted 10000 records. n: 568. Time: 45
Inserted 10000 records. n: 569. Time: 41
Inserted 10000 records. n: 570. Time: 39
Inserted 10000 records. n: 571. Time: 43
Inserted 10000 records. n: 572. Time: 68
Inserted 10000 records. n: 573. Time: 41
Inserted 10000 records. n: 574. Time: 33
Inserted 10000 records. n: 575. Time: 54
Inserted 10000 records. n: 576. Time: 39
Inserted 10000 records. n: 577. Time: 45
Inserted 10000 records. n: 578. Time: 50
Inserted 10000 records. n: 579. Time: 34
Inserted 10000 records. n: 580. Time: 36
Inserted 10000 records. n: 581. Time: 47
Inserted 10000 records. n: 582. Time: 59
Inserted 10000 records. n: 583. Time: 40
Inserted 10000 records. n: 584. Time: 37
Inserted 10000 records. n: 585. Time: 35
Inserted 10000 records. n: 586. Time: 59
Inserted 10000 records. n: 587. Time: 62
Inserted 10000 records. n: 588. Time: 26
Inserted 10000 records. n: 589. Time: 54
Inserted 10000 records. n: 590. Time: 54
Inserted 10000 records. n: 591. Time: 47
Inserted 10000 records. n: 592. Time: 53
Inserted 10000 records. n: 593. Time: 41
Inserted 10000 records. n: 594. Time: 45
Inserted 10000 records. n: 595. Time: 39
Inserted 10000 records. n: 596. Time: 45
Inserted 10000 records. n: 597. Time: 55
Inserted 10000 records. n: 598. Time: 50
Inserted 10000 records. n: 599. Time: 47
Inserted 10000 records. n: 600. Time: 39
Inserted 10000 records. n: 601. Time: 48
Inserted 10000 records. n: 602. Time: 43
Inserted 10000 records. n: 603. Time: 57
Inserted 10000 records. n: 604. Time: 38
Inserted 10000 records. n: 605. Time: 42
Inserted 10000 records. n: 606. Time: 44
Inserted 10000 records. n: 607. Time: 55
Inserted 10000 records. n: 608. Time: 46
Inserted 10000 records. n: 609. Time: 49
Inserted 10000 records. n: 610. Time: 44
Inserted 10000 records. n: 611. Time: 43
Inserted 10000 records. n: 612. Time: 43
Inserted 10000 records. n: 613. Time: 48
Inserted 10000 records. n: 614. Time: 36
Inserted 10000 records. n: 615. Time: 55
Inserted 10000 records. n: 616. Time: 58
Inserted 10000 records. n: 617. Time: 39

Is there any way to keep the performance linear at around 9-10 seconds per 10000 records?

Answer 1

Your calculations concentrate on the key side (Will the keys fit into RAM?). That is important, but after reaching some database size (which is dominated by the value side in your case), the values written to disk will become more relevant for speed. I fear that you can only optimize your disk (any details?) in such a situation.

Answer 2

Did you modify msiz too?

Did you try a lower bnum parameter (something like 1 or 2 Billions)?

What is your filesystem (ext4 performed far better than ext3 during my tests)?