如何在没有hadoop的情况下使用Hive

Question

I am a new to NoSQL solutions and want to play with Hive. But installing HDFS/Hadoop takes a lot of resources and time (maybe without experience but I got no time to do this).

Are there ways to install and use Hive on a local machine without HDFS/Hadoop?

Answer 1

yes you can run hive without hadoop 1.create your warehouse on your local system 2. give default fs as file:/// than you can run hive in local mode with out hadoop installation

In Hive-site.xml

<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> 
<configuration>
      <property>
         <name>hive.metastore.schema.verification</name> 
         <value>false</value> 
      </property> 
     <property> 
      <!-- this should eventually be deprecated since the metastore should supply this --> 
        <name>hive.metastore.warehouse.dir</name> 
        <value>file:///tmp</value>
        <description></description> 
     </property>
     <property> 
        <name>fs.default.name</name> 
        <value>file:///tmp</value> 
     </property> 
</configuration>

Answer 2

I would recomend you to use something like this.

http://hortonworks.com/products/hortonworks-sandbox/

Its a fully functional VM with everything you need to start right away.

Answer 3

If you are just talking about experiencing Hive before making a decision you can just use a preconfigured VM as @Maltram suggested (Hortonworks, Cloudera, IBM and others all offer such VMs)

What you should keep in mind that you will not be able to use Hive in production without Hadoop and HDFS so if it is a problem for you, you should consider alternatives to Hive

Answer 4

You cant, just download Hive, and run:

./bin/hiveserver2                                                                                                                                        
Cannot find hadoop installation: $HADOOP_HOME or $HADOOP_PREFIX must be set or hadoop must be in the path

Hadoop is like a core, and Hive need some library from it.

Answer 5

Update This answer is out-of-date : with Hive on Spark it is no longer necessary to have hdfs support.

---

Hive requires hdfs and map/reduce so you will need them. The other answer has some merit in the sense of recommending a simple / pre-configured means of getting all of the components there for you.

But the gist of it is: hive needs hadoop and m/r so in some degree you will need to deal with it.

Answer 6

Although, there are some details that you have to keep in mind it's completely normal to use Hive without HDFS. There are a few details one should keep in mind.

1. As a few commenters mentioned above you'll still need some .jar files from hadoop common .
2. As of today(XII 2020) it's difficult to run Hive/hadoop3 pair. Use stable hadoop2 with Hive2.
3. Make sure POSIX permissions are set correctly, so your local hive can access warehouse and eventually derby database location.
4. Initialize your database by manual call to schematool

You can use site.xml file pointing to local POSIX filesystem, but you can also set those options in HIVE_OPTS environmen variable. I covered that with examples of errors I've seen on my blog post

Answer 7

Top answer works for me. But need few more setups. I spend a quite some time search around to fix multiple problems until I finally set it up. Here I summarize the steps from scratch:

• Download hive, decompress it
• Download hadoop, decompress it, put it in the same parent folder as hive
• Setup hive-env.sh

   $ cd hive/conf $ cp hive-env.sh.template hive-env.sh

  Add following environment in hive-env.sh (change path accordingly based on actual java/hadoop version)

   JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_281.jdk/Contents/Home export path=$JAVA_HOME/bin:$path export HADOOP_HOME=${bin}/../../hadoop-3.3.1

• setup hive-site.xml

   $ cd hive/conf $ cp hive-default.xml.template hive-site.xml

  Replace all the variable ${system:***} with constant paths (Not sure why this is not recognized in my system). Set database path to local with following attributes (copied from top answer)

   <configuration> <property> <name>hive.metastore.schema.verification</name> <value>false</value> </property> <property> <.-- this should eventually be deprecated since the metastore should supply this --> <name>hive.metastore.warehouse:dir</name> <value>file.///tmp</value> <description></description> </property> <property> <name>fs.default:name</name> <value>file:///tmp</value> </property> </configuration>

• setup hive-log4j2.properties (optional, good for troubleshooting)

   cp hive-log4j2.properties.template hive-log4j2.properties

  Replace all the variable ${sys:***} to constant path
• Setup metastore_db If you directly run hive , when do any DDL, you will got error of:

   FAILED: HiveException org.apache.hadoop.hive.ql.metadata.HiveException:MetaException(message:Hive metastore database is not initialized. Please use schematool (eg ./schematool -initSchema -dbType...) to create the schema. If needed, don't forget to include the option to auto-create the underlying database in your JDBC connection string (eg ? createDatabaseIfNotExist=true for mysql))

  In that case we need to recreate metastore_db with following command

   $ cd hive/bin $ rm -rf metastore_db $./schematool -initSchema -dbType derby

• Start hive

   $ cd hive/bin $./hive

Now you should be able run hive on you local file system. One thing to note, the metastore_db will always be created on you current directory. If you start hive in a different directory, you need to recreate it again.