Difference between hive.tez.container.size and tez.task.resource.memory.mb

Question

Would someone know and explain to me please the difference between these settings of Tez ?

hive.tez.container.size and tez.task.resource.memory.mb

thanks.

Answer 1

hive.tez.container.size

This property specifies tez container size. Usually value of this property should be the same as or a small multiple (1 or 2 times that) of YARN container size yarn.scheduler.minimum-allocation-mb and should not exceed value of yarn.scheduler.maximum-allocation-mb .

As a general rule don't put value higher than memory per processor as you want 1 processor per container and you want to spun up multiple containers.

You can find very detailed answer and a great architecture diagram on Hortonworks community answer here

tez.task.resource.memory.mb

Amount of memory used by launched task in TEZ container. Usually this value is being set in DAG but if it's not then by default it will get set to 1024 MB.

All task in the DAG will have same size, it is helpful for container reuse and good for performance too.