使用sqoop（1.4.5）将mysql数据导入到hdfs（hadoop 2.4.1）

Question

I am a begineer in Hadoop, trying to import a table from my local database mysql, to hdfs of hadoop 2.4.1 using sqoop 1.4.5.

mysql version : 5.5.38   
hdfs version :hadoop 2.4.1   
sqoop version :1.4.5

i try sqoop-list-databases ,it give correct answer but then on giving following cmd:

sqoop import --connect jdbc:mysql://localhost:3306/information_schema --username root --password root --table VIEWs -m 1

and i get:

 ERROR tool.ImportTool: Encountered IOException running import job: java.io.FileNotFoundException: File does not exist: hdfs://localhost:54310/usr/lib/sqoop/lib/jackson-mapper-asl-1.9.13.jar
    at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1128)
    at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1120)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1120)
    at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)
    at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:224)
    at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:93)
    at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:57)
    at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:265)
    at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:301)
    at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:389)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
    at org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:186)
    at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:159)
    at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:247)
    at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:665)
    at org.apache.sqoop.manager.MySQLManager.importTable(MySQLManager.java:118)
    at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:497)
    at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:601)
    at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
    at org.apache.sqoop.Sqoop.main(Sqoop.java:236)

Answer 1

Your mapred-site.xml file is configured wrongly. Check the file content, The content should looks like shown below:

 <configuration>
 <property>
 <name>mapreduce.framework.name</name>
 <value>yarn</value>
</property>
</configuration>

Generally mapred-site.xml file content is made the same as core-site.xml content. Core-site.xml file content should looks like shown below:

<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://master:54310</value>
</property>
</configuration>

Tayfun YALCINKAYA

Answer 2

将MySQL驱动程序jar和所有缺少的jar添加到sqoop安装的lib文件夹中，以便能够使用mysql。

Answer 3

Please remove the temp file in HDFS (if some error like /user/hduser/*** is already exists).

1. copy sqoop from local to HDFS:

hadoop fs -mkdir -p /usr/lib/sqoop/

hadoop fs -copyFromLocal /usr/lib/sqoop/*.jar /usr/lib/sqoop/lib/

2. copy jar/class from tmp to HDFS

cp /tmp/sqoop-hduser/compile/* /usr/lib/sqoop/bin/

hadoop fs -put /tmp/sqoop-hduser/compile/* /user/hduser/