如何使用sqoop将mySql服务器中的JSON字段数据导入到Hive表中

Question

I have a table name EmpData in MySQL server with two field as EmpID (INT) and Details (nvarchar) . Details field contain JSON string like

{ "name": "Michel","address":{"house no":"12","street":"Johnson road","city":"New Delhi","country":"INDIA"}}

{ "name": "John","address":{"house no":"99","street":"Johnson road","city":"London","country":"UK"}}

I want to import Details field JSON data into hive table EmpHiveStore and query on hive table like

SELECT name,address.street from EmpHiveStore;

Is there any way to import JSON field data into Hive table using sqoop?

Thank You.

Answer 1

Sqoop's job here is to import data from RDBMS( MySQL in your case ) to Hive.

You need to use sqoop import command for this

sqoop import \
--connect jdbc:mysql://mysql.example.com/testdb \
--username root \
--password root \
--table EmpData \
--columns Details \
--hive-import \
--hive-table EmpHiveStore

This will create hive table if not exists. Sqoop's job is done here . Hive will store this data in the String. There is no special JSON type in Hive. So, you won't be able to perform query like

SELECT name,address.street from EmpHiveStore;

directly via hive.

You can get the location of data using

show create table EmpHiveStore;

You have only 1 column in your table and it has JSON data. So it's basically a JSON file in HDFS.

You want to query into nested JSON. You need another tool that allows you to perform queries like this. You can explore SQL query engine like Drill and Spark SQL for this.