简体繁体中英

Process UTF-16LE encoded file in hadoop/cascading

原文 2018-02-08 09:49:28 8 1 java/ hadoop/ mapreduce/ cascading/ utf-16le

I need to process a UTF-16LE encoded file in cascading on top of hadoop. I've tried following approaches but none of these are working.

While assigning value -Xmx1024m -Dfile.encoding=UTF-16LE to property mapreduce.map.java.opts in mapred-site.xml failed due to NullPointerException at: com.google.common.base.Preconditions.checkNotNull(Preconditions.java:187) But this method works for UTF-8. Is hadoop unable to process UTF-16 data?
Doing System.setProperty("file.encoding", "UTF-16LE"); in code is also unable to parse the data
Overriding charset of TextDelimited class of Cascading is also unable to process data

However using BufferedReader to read it in UTF-16LE parses the data correctly.

Please help

Thanks in advance

1 answers

在某处发现：Hadoop不支持UTF-16文件

How to deal with UTF-16LE encoded text file using Java? or convert it to ASCII?

opening xls file and saving it as tsv file using java and UTF-16LE to UTF-8 conversion

Java Parsing XML from UTF-16LE string

UTF-16LE encoding and xerces2 Java

Fast conversion of String to byte[] using UTF-16LE encoding

Java UTF-16LE Weird Error That Stumped My Professor

Unable to use encoding UTF-16LE on Android Version 9

Reading and Writing Text files with UTF-16LE encoding and Apache Commons IO

How do I encode/decode UTF-16LE byte arrays with a BOM?

How to validate and parse an UTF-16 encoded XML file in Java?

暂无

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

Related Question How to deal with UTF-16LE encoded text file using Java? or convert it to ASCII? opening xls file and saving it as tsv file using java and UTF-16LE to UTF-8 conversion Java Parsing XML from UTF-16LE string UTF-16LE encoding and xerces2 Java Fast conversion of String to byte[] using UTF-16LE encoding Java UTF-16LE Weird Error That Stumped My Professor Unable to use encoding UTF-16LE on Android Version 9 Reading and Writing Text files with UTF-16LE encoding and Apache Commons IO How do I encode/decode UTF-16LE byte arrays with a BOM? How to validate and parse an UTF-16 encoded XML file in Java?

Related Tags

Process UTF-16LE encoded file in hadoop/cascading

Question

1 answers

solution1 0 2018-05-03 02:59:19

solution1
0 2018-05-03 02:59:19