Reclaim free space from MongoDB 3.2

Question

Background: MongoDB ReplicaSet with 1 member and 1 arbiter is set up. It's necessary to free up disk space, because data in MongoDB is no more needed. All documents were removed with db.collection.remove({}) command. After db.runCommand ( { compact: 'collection', force:true} ) , db.collection.reIndex() and db.repairDatabase() were ran, db.stats() looked like:

> db.stats()
{
    "db" : "database",
    "collections" : 2,
    "objects" : 6,
    "avgObjSize" : 167.83333333333334,
    "dataSize" : 1007,
    "storageSize" : 153042944,
    "numExtents" : 0,
    "indexes" : 5,
    "indexSize" : 45056,
    "ok" : 1
}

As you can see, there are just 6 documents in DB (in the second collections) with total storage size of 153MB. But, if we look at the disk space usage:

# du -h --max-depth=1 | grep mongodb
9.8G    ./mongodb

And inside mongodb directory:

ls -l
total 9365696
-rw-r----- 1 username usergroup      16384 Aug 19 14:19 collection-0-2472884588219438804.wt
-rw-r----- 1 username usergroup      36864 Aug 19 14:19 collection-0-4266045208498277842.wt
-rw-r----- 1 username usergroup      12288 Aug 24 15:55 collection-0--7009505821458556818.wt
-rw-r----- 1 username usergroup      49152 Aug 19 14:20 collection-0--7439052959034576211.wt
-rw-r----- 1 username usergroup      36864 Aug 19 14:20 collection-0-8676027872146699793.wt
-rw-r----- 1 username usergroup      36864 Aug 19 14:19 collection-2-2472884588219438804.wt
-rw-r----- 1 username usergroup      32768 Aug 23 13:54 collection-2-2911328926458913167.wt
-rw-r----- 1 username usergroup 9589645312 Aug 24 15:55 collection-4-2472884588219438804.wt
-rw-r----- 1 username usergroup      36864 Aug 19 14:19 collection-7--7439052959034576211.wt
-rw-r----- 1 username usergroup      36864 Aug 19 14:19 collection-9--7439052959034576211.wt
drwxr-x--- 2 username usergroup       4096 Aug 24 16:03 diagnostic.data
-rw-r----- 1 username usergroup       4096 Aug 24 15:54 index-10-561247476684508201.wt
-rw-r----- 1 username usergroup      36864 Jul 20 15:09 index-10--7439052959034576211.wt
-rw-r----- 1 username usergroup      36864 Aug 19 14:19 index-11--7439052959034576211.wt
-rw-r----- 1 username usergroup      16384 Mar  4 09:20 index-1-2472884588219438804.wt
-rw-r----- 1 username usergroup      36864 Jul 20 15:09 index-1-4266045208498277842.wt
-rw-r----- 1 username usergroup      36864 Aug 19 14:20 index-1--7439052959034576211.wt
-rw-r----- 1 username usergroup      16384 Jul  5 13:25 index-1-8676027872146699793.wt
-rw-r----- 1 username usergroup      36864 Jul 20 15:09 index-2-4266045208498277842.wt
-rw-r----- 1 username usergroup      16384 Mar  4 09:20 index-3-2472884588219438804.wt
-rw-r----- 1 username usergroup      16384 Aug 23 14:00 index-6-561247476684508201.wt
-rw-r----- 1 username usergroup      16384 Aug 23 14:00 index-7-561247476684508201.wt
-rw-r----- 1 username usergroup       4096 Aug 24 15:54 index-8-561247476684508201.wt
-rw-r----- 1 username usergroup      16384 Aug 19 14:19 index-8--7439052959034576211.wt
-rw-r----- 1 username usergroup       4096 Aug 24 15:54 index-9-561247476684508201.wt
drwxr-x--- 2 username usergroup       4096 Aug 24 15:55 journal
-rw-r----- 1 username usergroup      36864 Aug 24 15:55 _mdb_catalog.wt
-rwxr-x--- 1 username usergroup          6 Aug 19 14:19 mongod.lock
-rw-r----- 1 username usergroup      36864 Aug 24 15:56 sizeStorer.wt
-rw-r----- 1 username usergroup         95 Dec 30  2015 storage.bson
-rw-r----- 1 username usergroup         46 Dec 30  2015 WiredTiger
-rw-r----- 1 username usergroup        534 Dec 30  2015 WiredTiger.basecfg
-rw-r----- 1 username usergroup       4096 Aug 19 14:19 WiredTigerLAS.wt
-rw-r----- 1 username usergroup         21 Dec 30  2015 WiredTiger.lock
-rw-r----- 1 username usergroup        903 Aug 24 15:57 WiredTiger.turtle
-rw-r----- 1 username usergroup      94208 Aug 24 15:57 WiredTiger.wt

This string seems strange to me:

    -rw-r----- 1 username usergroup 9589645312 Aug 24 15:55 collection-4-2472884588219438804.wt

Question: What can be in mystery collection-4-2472884588219438804.wt file? Why is it not affected by compact, repairDatabase commands? Is there any way to force MongoDB empty this file or somehow reclaim it's space?

Update: With help of @james-wahlin we've figured out, that 9.8Gb is the Replica Set Oplog's size. But how can I force MongoDB to free up space, despite potential data loss for other Replica Set Members?

Answer 1

Based on the fact you removed all documents and the data is no longer needed:

1. Stop the mongod process

2. Remove all files from the ./mongodb directory manually

    > use admin switched to db admin > db.shutdownServer() 2016-08-24T08:50:29.085-0500 I NETWORK DBClientCursor::init call() failed server should be down... 2016-08-24T08:50:29.096-0500 I NETWORK trying reconnect to 127.0.0.1:27017 (127.0.0.1) failed 2016-08-24T08:50:30.103-0500 W NETWORK Failed to connect to 127.0.0.1:27017, reason: errno:10061 No connection could be made because the target machine actively refused it. 2016-08-24T08:50:30.104-0500 I NETWORK reconnect 127.0.0.1:27017 (127.0.0.1) failed failed couldn't connect to server 127.0.0.1:27017 (127.0.0.1), connection attempt failed > # cd ./mongodb # rm -rf * 

When you restart mongo it will initialize the mongodb directory as needed.