sqoop增量导入更新的行和新创建的记录

Question

I have the sqoop job created with incremental append with last value

Job:

sqoop job --create myjob2 -- import --connect jdbc:mysql://host/DBnam -username user -password passwor --table savingssmal --check-column id --incremental append --last-value 0 --target-dir /user/xxxx/prac/sqoop --split-by id --as-parquetfile -m 1

My question is: I want to import the newly created record and also updated record into a mysql table?

Can you please help me in this?

Answer 1

You can use the lastmodified mode for incremental Sqoop import.

The append mode (used in your example) is used to import rows based on increasing row id values. So, when the job runs, it will import rows where --check-column (ie id ) is greater than --last-value (ie 0 ). If a row is updated, the id will generally remain the same and the updated row won't be imported.

The lastmodified mode is used to import rows based on a timestamp column (for example, last_modified_time ). When the job runs, it will import rows where --check-column is more recent than specified via --last-value . The application writing to the table should update the last_modified_time column on inserts and updates. This way, both newly inserted and updated rows will be imported when the Sqoop job runs.

A sample invocation based on your example with the lastmodified mode would be as below:

sqoop job --create myjob2 -- import --connect jdbc:mysql://host/DBnam -username user -password passwor --table savingssmal --check-column last_update_time --incremental lastmodified --last-value "2018-02-03 04:38:39.0" --target-dir /user/xxxx/prac/sqoop --as-parquetfile -m 1