Ceph storage usable space calculation

Question

Can some help me with below question.

How I can calculate total usable ceph storage space. Lets say I have 3 nodes and each nodes has 6 OSD of 1TB disk . That is total of 18TB storage ( 3* 6TB ) .All these 18TB space is usable or some space will go for redundancy ?

Answer 1

Ceph has two important values: full and near-full ratios. Default for full is 95% and nearfull is 85%. ( http://docs.ceph.com/docs/jewel/rados/configuration/mon-config-ref/ )

If any OSD hits the full ratio it will stop accepting new write requrests (Read: you cluster stucks). You can raise this value, but be careful, because if OSD stops because there is no space left (at FS level), you may experience data loss.

That means, that you couldn't get more than full ratio out of your cluster, and for normal operations it's wise to not reach nearfull value.

For you case, with redundancy 3, you have 6*3 Tb of raw space, this translates to 6 TB of protected space, after multiplying by 0.85 you have 5.1Tb of normally usable space.

Two more unsolicited advises: Use at least 4 nodes (3 is a bare minimum to work, if one node is down, you have a trouble), and use lower values for near-full. I'd advice to have it around 0.7. In this case you will have (4 nodes, 6 * 1Tb OSD, /3, *.7) 5.6 Tb of usable space.