我们如何在开源Spark和Hortonworks的Hadoop沙箱中使用集群？

Question

I have a conceptual question. I downloaded Apache Spark and Hortonworks Hadoop Sandbox. As far as I know, we analyze big data by distributing the tasks to multiple machines or clusters. Amazon Web Services provide customers clusters when they pay for their services. But in the case of Spark or Hadoop, whose clusters I am using when I simply download these environments? They say that these environments provide a single-node clusters, which is, I assume my computer itself. But then, how can I analyze big data if I am limited to my computer itself? In brief, what is the logic of using Spark on my own laptop?

Answer 1

The environments are exactly what they say they are, a sandbox. It can be used to test functionality but not performance because as you rightly said, they are running out of your laptop. The VM comes configured with all the software neccesary for you to test exactly this.

If you wish to get the true performance potential of spark, then you will need to install spark on a cluster of servers using the procedures that they describe here and then you will be truly using the computational power from the servers that you just installed spark on.

Hope that helps!