使用 Java 从 Hive 表中读取记录（通过 Hivemetastoreclient 或 Hcatalog 或 WebHcat）

Question

One Hive table t_event is in demo_read database. Table has more than 100,000 records.How to read records through java API.

Answer 1

You can use Hive JDBC driver to connect to Hive tables. It's okay for testing or POC with the code below but I recommend moving your end tables to HBase (check Phoenix) or MongoDB or some sort of Relational based table which have low latency.

You could as well use dynamic partitions or some sort of cluster technique in Hive for better performance. You can use the following code, I haven't tested it (use it as a sample).

 import java.sql.*;

 public class HiveDB {
 
 public static final String HIVE_JDBC_DRIVER = "org.apache.hadoop.hive.jdbc.HiveDriver";
 public static final String HIVE_JDBC_EMBEDDED_CONNECTION = "jdbc:hive://";
 private static String driverName = "org.apache.hadoop.hive.jdbc.HiveDriver";
 
 private Statement getConnection() throws ClassNotFoundException,
            SQLException {
        Class.forName(HIVE_JDBC_DRIVER);
        Connection connection = DriverManager.getConnection(
                HIVE_JDBC_EMBEDDED_CONNECTION, "", "");

        Statement statement = connection.createStatement();
        return statement;
    }
    
    public static void main(String[] args) {
       HiveDB hiveDB = new HiveDB();
       try {
         Statement statement = hiveDB.getConnection();
         //print each row
         ResultSet resultSet = statement.executeQuery("select * from demo_read.t_event");
         int columns = resultSet.getMetaData().getColumnCount();
         while (resultSet.next()) {
           for ( int i = 0 ; i < columns; ++i) {
              System.out.print(resultSet.getString(i + 1) + " " );
              if (i == 100) break; //print up to 100th rows
           }
           System.out.println();
         }
        statement.close(); //close statement
      } catch (ClassNotFoundException e) {
         //
      } catch (SQLException e) {
         //
      }
    }
    
 }

Answer 2

Well, actually you don't want to read all that data. You need to transform it and upload into some database or (if data is relatively small) to export it into common format (CSV, JSON, etc.).

You could transform data with Hive CLI, WebHCat or JDBC Hive driver.