Sqoop 导入错误 Error: java.io.IOException: SQLException in nextKeyValue

Question

I have written a script to perform an incremental import of data from oracle table to HDFS directory. I use the following sqoop command to do the import :

    sqoop -- import \
   --connect $JDBCconnectionString  \
   --username $dbUserName \
   --password-file $passwordLocal \
   --query 'select * from dmt_sim.dim_product WHERE $CONDITIONS' \
   --split-by "PRODUCT_TITLE" \
   --incremental append \
   --check-column "KEY" \
   --last-value "1" \
   --append \
   --fields-terminated-by '\t' \
   --target-dir /user/ksrinivasan/dmn_product

The values for the variables $JDBCconnectionString,$dbUserName, $passwordLocal is substituted during run so the connection to the oracle database and getting the boundary values all succeeds but when the job is started it throws an error.

16/01/25 06:19:29 INFO mapreduce.Job: Job job_1452256584707_106782 running in uber mode : false
06:19:36            16/01/25 06:19:29 INFO mapreduce.Job:  map 0% reduce 0%
06:20:07            16/01/25 06:20:00 INFO mapreduce.Job: Task Id : attempt_1452256584707_106782_m_000000_0, Status : FAILED
06:20:07            Error: java.io.IOException: SQLException in nextKeyValue
06:20:07                at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:277)
06:20:07                at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:553)
06:20:07                at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
06:20:07                at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
06:20:07                at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
06:20:07                at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
06:20:07                at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
06:20:07                at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
06:20:07                at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
06:20:07                at java.security.AccessController.doPrivileged(Native Method)
06:20:07                at javax.security.auth.Subject.doAs(Subject.java:415)
06:20:07                at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
06:20:07                at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
06:20:07            Caused by: java.sql.SQLSyntaxErrorException: ORA-00907: missing right parenthesis
06:20:07            
06:20:07                at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:445)
06:20:07                at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396)
06:20:07                at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:879)
06:20:07                at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:450)
06:20:07                at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
06:20:07                at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)
06:20:07                at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207)
06:20:07                at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:884)
06:20:07                at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1167)
06:20:07                at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1289)
06:20:07                at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3584)
06:20:07                at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3628)
06:20:07                at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1493)
06:20:07                at org.apache.sqoop.mapreduce.db.DBRecordReader.executeQuery(DBRecordReader.java:111)
06:20:07                at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:235)
06:20:07                ... 12 more
06:20:07            

So it nice if someone else has faced the same problem and they have some insight about how to debug this kinda issues ???

Answer 1

nextKeyValue 中的 SQLException 这意味着在您执行时，源 KEY 列中没有最后一个值“1”。

Answer 2

Check for all the possible column's values as null and specially check for date column that may have date as '0000-00-00' and you were using in select clause of dmt_sim.dim_product.

Identify the columns and apply if condition in select to replace null or date with something else(that you would able to recognized).

Answer 3

It´s because the column used to split is not a numeric ("PRODUCT_TITLE").

When Sqoop translate the query fails at give the intervals "PRODUCT_TITLE" >= xx <= yy

trait with a numeric field.

When execute the sentences include --verbose 2> file_log.log (it´s to see if the split sentences are correct)