纱线-让hadoop使用更多资源

Question

As you can see, this node contains 3 containers (which is using 6GB memory and 3 vCores). I would like it to use rest of the vCores ( 5 vCores in vCores Avail column). I've not done any configuration on the yarn-site.xml yet.

Answer 1

Yarn uses only the resources that it needs

What you are currently looking at seems to be the resources currently used by the running jobs.

Yarn knows that it is allowed to use 2GB more memory and 5 more vcores, but it seems that by the nature of the job these can simply not be utilized.

Hence, it is not likely a problem, or somethign that needs to be fixed but just a matter of you running job of this nature.

---

Example

When I say a job of this nature, I mean a job for which it is not required to use 3 containers of 2 GB each.

The most simple example of such a job would be a count of 3 comparatively small files with a default container size of 2 GB.

If you really would want to run this kind of job more in parrallel, you would need to resort to workarounds (like setting a tiny maximum container size, or splitting all files in half). However, I would not recommend this.