是否可以使用 HADOOP YARN 运行任何应用程序或程序？

Question

I'm studying distributed computing recently and found out Hadoop Yarn is one of them. So thought if I just establish Hadoop Yarn cluster, then every application will run distributed.

But now someone told me that HADOOP Yarn cannot do anything by itself and need other things like mapreduce, spark, and hbase.

If this is correct, then is that mean only limited tasks can be run with Yarn? Or can I apply Yarn's distributed computing to all applications I want?

Answer 1

Hadoop is the name which refers to the entire system.

HDFS is the actual storage system. Think of it as S3 or a distributed Linux filesystem.

YARN is a framework for scheduling jobs and allocating resources. It handles these things for you, but you don't interact very much with it.

Spark and MapReduce are managed by Yarn. With these two, you can actually write your code/applications and give work to the cluster.

HBase uses the HDFS storage (with is file based) and provides NoSql storage.

Theoretically you can run more than just Spark and MapReduce on Yarn and you can use something else then Yarn (Kubernetes is in works or supported now). You can even write your own processing tool, queue/resource management system, storage... Hadoop has many pieces which you may use or not, depending on your case. But the majority of Hadoop systems use Yarn and Spark.

If you want to deploy Docker containers for example, just a Kubernetes cluster would be a better choice. If you need batch/real time processing with Spark, use Hadoop.

Answer 2

YARN itself is a resource manager. You will need to write code that can be deployed onto those resources, and then that could do anything, given that the nodes running the tasks are themselves capable of running the job. For example, you cannot distribute a Python library without first installing the dependencies for that script. Mesos is a bit more generalized / accessible than YARN, if you want more flexibility for the same affect.

YARN mostly supports running JAR files, shell scripts (at least, from Oozie) or Docker containers can be deployed to it as well (refer Apache docs)

You may also refer to the Apache Slider or Twill projects for more information.