How to increase the space for dfs on HDFS cluster

Question

We have 4 datanode HDFS cluster ...there is large amount of space avialable on each data node of about 98gb ...but when i look at the datanode information .. it's only using about 10gb ...

How can we make it use all the 98gb and not run out of space as indicated in image

this is the hdfs-site.xml on name node

<configuration>

    <property>
        <name>dfs.replication</name>
        <value>2</value>
    </property>

    <property>
        <name>dfs.name.dir</name>
        <value>file:///test/hadoop/hadoopinfra/hdfs/namenode</value>
    </property>

    <property>
        <name>hadoop.tmp.dir</name>
        <value>file:///tmp/hadoop/data</value>
    </property>

    <property>
        <name>dfs.datanode.du.reserved</name>
        <value>2368709120</value>
    </property>

    <property>
        <name>dfs.datanode.fsdataset.volume.choosing.policy</name>
        <value>org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy</value>
    </property>

    <property>
        <name>dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction</name>
        <value>1.0</value>
    </property>

</configuration>

this is the hdfs-site.xml under data node

<configuration>

    <property>
        <name>dfs.replication</name>
        <value>2</value>
    </property>

    <property>
        <name>dfs.data.dir</name>
        <value>file:///test/hadoop/hadoopinfra/hdfs/datanode</value>
    </property>

    <property>
        <name>hadoop.tmp.dir</name>
        <value>file:///tmp/hadoop/data</value>
    </property>

    <property>
        <name>dfs.datanode.du.reserved</name>
        <value>2368709120</value>
    </property>

    <property>
        <name>dfs.datanode.fsdataset.volume.choosing.policy</name>
        <value>org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy</value>
    </property>

    <property>
        <name>dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction</name>
        <value>1.0</value>
    </property>

</configuration>

the 98gb is under /test

Please let us know if we missed anything in the configuration

Answer 1

Look at the dfs.datanode.data.dir in the hdfs-site.xml. This property would control all the directories which can be used to store DFS blocks.

Documentation Link

So on you machines execute "df -h" that should list all the mount points which make up the 98 GB. Then in each of the mount points decide which directory can be used to store HDFS block data and add those under hdfs-site.xml comma separated for dfs.datanode.data.dir. Then retstart namenode and all data node services.

And from your edited post :

   <property>
        <name>dfs.data.dir</name>
        <value>file:///test/hadoop/hadoopinfra/hdfs/datanode</value>
    </property>

It should not be file:// . It should look like :

   <property>
        <name>dfs.data.dir</name>
        <value>/test/hadoop/hadoopinfra/hdfs/datanode</value>
    </property>

Same for other properties.