Java - Apache Beam：使用“UCS2-LE BOM”编码从 GCS 读取文件

Question

I want to read a file in UCS2-LE BOM using TextIO, however It doesn't seem to work. Is there a way to use TextIO with this encoding ? Or is there another library that does well with this type of encoding ?

My code is in JAVA (Apache Beam)

PCollection<KV<String, String>> csvElements =
            pipeline.apply("Reads the input csv file", TextIO
                    .read()
                    .from(options.getPolledFile()))
                    .apply("Read File", ParDo.of(new DoFn<String, KV<String,String>>(){
                        @ProcessElement
                        public void processElement(ProcessContext c) throws UnsupportedEncodingException {
                            String element = c.element();

                            String elStr = new String(element.getBytes(),"UTF-16LE");
                            c.output(elStr);}}));

Answer 1

I found a solution, in a medium post : Solution

The file I am reading is stored in GCS, hence the added lines in try part (compared to the original code.)

file = "path to gas file";    
PCollection<String> readCollection = pipeline.apply(FileIO.match().filepattern(file))
                    .apply(FileIO.readMatches())
                    .apply(FlatMapElements
                            .into(strings())
                            .via((FileIO.ReadableFile f) -> {
                                List<String> result = new ArrayList<>();
                                try {
                                    ReadableByteChannel byteChannelParse = f.open();
                                    InputStream inputStream = Channels.newInputStream(byteChannelParse);
                                    BufferedReader br = new BufferedReader(new InputStreamReader(inputStream, "UTF-16"));
                                    String line = br.readLine();
                                    while (line != null) {
                                        result.add(line);
                                        line = br.readLine();
                                    }
                                    br.close();
                                    inputStream.close();
                                }

                                catch (IOException e) {
                                    throw new RuntimeException("Error while reading", e);
                                }
                                return result;
                            }));

PS: I didn't add a line with credentials because I passed it into IntelliJ parameters.