[英]Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/RecordReader
I am trying to convert my Json file to Parquet format. 我正在尝试将我的Json文件转换为Parquet格式。
Following is my pom file. 以下是我的pom文件。
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.mypackage</groupId>
<artifactId>JSONToParquet</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<repositories>
<repository>
<id>wso2</id>
<url>http://dist.wso2.org/maven2/</url>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>org.kitesdk</groupId>
<artifactId>kite-data-core</artifactId>
<version>1.1.0</version>
</dependency>
<dependency>
<groupId>org.kitesdk</groupId>
<artifactId>kite-morphlines-all</artifactId>
<version>1.0.0</version> <!-- or whatever the latest version is -->
<type>pom</type>
</dependency>
<!-- https://mvnrepository.com/artifact/ua_parser/ua-parser -->
<dependency>
<groupId>ua_parser</groupId>
<artifactId>ua-parser</artifactId>
<version>1.3.0</version>
<type>pom</type>
</dependency>
</dependencies>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
</properties>
</project>
Following is the code for conversion : 以下是转换代码:
Schema jsonSchema = JsonUtil.inferSchema(inputstream, "Movie", 10);
try (JSONFileReader<Movie> reader = new JSONFileReader<>(
inputstream, jsonSchema, Movie.class)) {
reader.initialize();
ParquetWriter parquetWriter
= new AvroParquetWriter(outputPath, jsonSchema, compressionCodecName, ParquetWriter.DEFAULT_BLOCK_SIZE, ParquetWriter.DEFAULT_PAGE_SIZE);
for (Movie record : reader) {
parquetWriter.write(record);
}
In the above code Movie
is my POJO class. 在上面的代码中, Movie
是我的POJO类。
When I run the program I am facing the following exception : 当我运行程序时,我面临以下异常:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/RecordReader
at com.mypackage.jsontoparquet.JsonToParquet.main(JsonToParquet.java:34)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.RecordReader
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 1 more
I am using JDK : 8. 我正在使用JDK:8。
I don't have any background of hadoop, so I am unable to understand it's root cause. 我没有任何关于hadoop的背景知识,所以我无法理解它的根本原因。
What is the issue ? 有什么问题?
Based on Kite-SDK Documentation, JSONFileReader
, ParquetWriter
and AvroParquetWriter
use Hadoop libraries to work. 基于Kite-SDK文档, JSONFileReader
, ParquetWriter
和AvroParquetWriter
使用Hadoop库工作。 It is needed to add Hadoop dependencies in your pom. 需要在pom中添加Hadoop依赖项。 You need at least below dependencies. 您至少需要以下依赖项。 Add them in your pom.xml: 将它们添加到您的pom.xml中:
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>2.6.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.6.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-jobclient</artifactId>
<version>2.6.0</version>
</dependency>
Your kite is missing hadoop dependencies 您的风筝缺少hadoop依赖项
there are some cases where you may have to provide the relevant Hadoop component dependencies yourself, and Kite has grouping dependencies for this purpose. 在某些情况下,您可能必须自己提供相关的Hadoop组件依赖关系,而Kite为此提供了分组依赖关系。
For Haddop2 (default) add to your pom: 对于Haddop2(默认),将其添加到pom:
<dependency>
<groupId>org.kitesdk</groupId>
<artifactId>kite-hadoop2-dependencies</artifactId>
<version>1.0.0</version>
<type>pom</type>
<scope>compile</scope>
</dependency>
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.