Ceph 原始存储使用与池存储使用

Question

I am new user of Ceph.

My Ceph server is running version 12.2.5. It has 3 nodes and each node is having 1 OSD with 4TB HDD.

The ceph.conf

osd pool default size     = 3
osd pool default min size = 2

I have upload 252GB data to Ceph.

For my understanding, the raw storage usage should be around 3 times because of setting osd pool default size = 3 .

But now my global raw storage usage is almost 6 times of my data (1605GB / 252GB = 6.3)

GLOBAL:
    SIZE       AVAIL     RAW USED     %RAW USED
    11165G     9560G        1605G         14.38
POOLS:
    NAME                          ID     USED      %USED     MAX AVAIL     OBJECTS
    .rgw.root                     8       1113         0         3000G           4
    default.rgw.control           9          0         0         3000G           8
    default.rgw.meta              10     21502         0         3000G          72
    default.rgw.log               11         0         0         3000G         269
    default.rgw.buckets.index     15         0         0         3000G         427
    default.rgw.buckets.data      16      256G      7.88         3000G     6522963

Is anyone have the same problem on Ceph?

Update

I think I find out the reason. Most of my file is a small file, but in Ceph version 12.2.5 (bluestore), bluestore have minimum allocate size bluestore_min_alloc_size_hdd = 65536 , therefore if the file is small than 64k, it still use 64k to store it.

I tested upload 2000 files (each 64k) and 3200 files (each 40k), both total around 128M

The raw usage of 64k file is 410M, close to 384M (2000 * 64k * 3)

The raw usage of 40k file is 602M, close to 614M (3200 * 64k * 3)

Answer 1

I am having the same issue. I have only realised this after creating the pool. You can change the min_alloc to 4096 but I don't know how that will perform for you. I would advise though to go halfway to 32k or 16k but not lower.