如何将mysql表同步到配置单元表？ （不支持sqoop --incremental lastmodified配置单元导入）

Question

I want to sync a mysql table into hive table. Because records in orders table usually changed in nearly future . I need update them into hive .

For example ,

1. I dump all mysql data into hive
2. daily job check the changed record which time_update is in nearly 1 days, and update them into hive table.

I have tried --incremental lastmodified like below

sqoop import \
"-Dorg.apache.sqoop.splitter.allow_text_splitter=true" \
--connect $DB_URL \
--username $USERNAME \
--password $PASSWORD \
--direct \
--fields-terminated-by '\t' \
--target-dir '/data/hive/' \
--delete-target-dir \
--hive-database $HIVE_DB \
--hive-table $HIVE_TABLE \
--hive-import \
--hive-overwrite \
--create-hive-table \
--query 'select * from '$HIVE_TABLE' where $CONDITIONS' \
--split-by id \
-m 6 \
--merge-key id \
--incremental lastmodified \
--check-column time_update \
--last-value "2019-01-01 21:00:00"

Got error --incremental lastmodified option for hive imports is not supported. Please remove the parameter --incremental lastmodified. --incremental lastmodified option for hive imports is not supported. Please remove the parameter --incremental lastmodified.

What is the proper way to do without --incremental lastmodified option .

Answer 1

First, you have to remove --delete-target-dir and --create-hive-table arguments as in incremental import, the target dir will stay as it is so --delete-target-dir will not work with --incremental argument. Also, hive-table should be created once only so you have to remove --create-hive-table argument and create hive table manually in hive with same schema, take the location of that schema and use it as --target-dir.

sqoop import \
--connect <<db_url>> \
--username <<username>> \
--password <<password>> \
--direct \
--fields-terminated-by '\t' \
--hive-database <<hive_db>> \
--hive-table <<hive_table>> \
--hive-import \
--hive-overwrite \
--query 'select * from <<db_table>> where $CONDITIONS' \
--split-by product_id \
-m 6 \
--merge-key product_id \
--incremental lastmodified \
--check-column timedate \
--last-value 0 \
--target-dir /user/hive/warehouse/problem5.db/products_hive (<<hive_table_location>>)

This will work successfully, if not let me know.