CEPH原始空间使用

Question

I can't understand, where my ceph raw space is gone.

cluster 90dc9682-8f2c-4c8e-a589-13898965b974
     health HEALTH_WARN 72 pgs backfill; 26 pgs backfill_toofull; 51 pgs backfilling; 141 pgs stuck unclean; 5 requests are blocked > 32 sec; recovery 450170/8427917 objects degraded (5.341%); 5 near full osd(s)
     monmap e17: 3 mons at {enc18=192.168.100.40:6789/0,enc24=192.168.100.43:6789/0,enc26=192.168.100.44:6789/0}, election epoch 734, quorum 0,1,2 enc18,enc24,enc26
     osdmap e3326: 14 osds: 14 up, 14 in
      pgmap v5461448: 1152 pgs, 3 pools, 15252 GB data, 3831 kobjects
            31109 GB used, 7974 GB / 39084 GB avail
            450170/8427917 objects degraded (5.341%)
                  18 active+remapped+backfill_toofull
                1011 active+clean
                  64 active+remapped+wait_backfill
                   8 active+remapped+wait_backfill+backfill_toofull
                  51 active+remapped+backfilling
recovery io 58806 kB/s, 14 objects/s

OSD tree (each host has 2 OSD):

# id    weight  type name       up/down reweight
-1      36.45   root default
-2      5.44            host enc26
0       2.72                    osd.0   up      1
1       2.72                    osd.1   up      0.8227
-3      3.71            host enc24
2       0.99                    osd.2   up      1
3       2.72                    osd.3   up      1
-4      5.46            host enc22
4       2.73                    osd.4   up      0.8
5       2.73                    osd.5   up      1
-5      5.46            host enc18
6       2.73                    osd.6   up      1
7       2.73                    osd.7   up      1
-6      5.46            host enc20
9       2.73                    osd.9   up      0.8
8       2.73                    osd.8   up      1
-7      0               host enc28
-8      5.46            host archives
12      2.73                    osd.12  up      1
13      2.73                    osd.13  up      1
-9      5.46            host enc27
10      2.73                    osd.10  up      1
11      2.73                    osd.11  up      1

Real usage:

/dev/rbd0        14T  7.9T  5.5T  59% /mnt/ceph

Pool size:

osd pool default size = 2

Pools: ceph osd lspools

0 data,1 metadata,2 rbd,

rados df

pool name       category                 KB      objects       clones     degraded      unfound           rd        rd KB           wr        wr KB
data            -                          0            0            0            0           0            0            0            0            0
metadata        -                          0            0            0            0           0            0            0            0            0
rbd             -                15993591918      3923880            0       444545           0        82936      1373339      2711424    849398218
  total used     32631712348      3923880
  total avail     8351008324
  total space    40982720672

Raw usage is 4x real usage. As I understand, it must be 2x ?

Answer 1

Yes, it must be 2x. I don't really shure, that the real raw usage is 7.9T. Why do you check this value on mapped disk?

This are my pools:

pool name                 KB      objects       clones     degraded      unfound           rd        rd KB           wr        wr KB
admin-pack           7689982         1955            0            0            0       693841      3231750     40068930    353462603
public-cloud       105432663        26561            0            0            0     13001298    638035025    222540884   3740413431
rbdkvm_sata      32624026697      7968550        31783            0            0   4950258575 232374308589  12772302818 278106113879
  total used     98289353680      7997066
  total avail    34474223648
  total space   132763577328

You can see, that the total amount of used space is 3 times more than the used space in the pool rbdkvm_sata (+-).

ceph -s shows the same result too:

pgmap v11303091: 5376 pgs, 3 pools, 31220 GB data, 7809 kobjects
            93736 GB used, 32876 GB / 123 TB avail

Answer 2

I don't think you have just one rbd image. The result of "ceph osd lspools" indicated that you had 3 pools and one of pools had name "metadata".(Maybe you were using cephfs). /dev/rbd0 was appeared because you mapped the image but you could have other images also. To list the images you can use "rbd list -p ". You can see the image info with "rbd info -p "