非UTF 8编码的CSV特殊字符问题

Question

Hi I am using a spring mvc application to process excel and csv file. I have encountered one issue that for special characters such as DèéêàáâÉ once process it is converting it to D which is wrong. However when the csv file encoding is UTF8 the special characters is converted successfully.

Part of the ajax call is shown below:

$('#fileuploading').fileupload({

url: 'uploadFile',
dataType: 'json',
acceptFileTypes: /(\.|\/)(csv|xlsx)$/i,
maxFileSize: 10000000,
autoUpload: false,
 disableImageLoad: true,
disableAudioPreview: true,
disableVideoPreview: true,
disableValidation: false,
disableImageResize: true

})

My controller method is shown below:

@RequestMapping(value = "/uploadFile", method = RequestMethod.POST)
    public @ResponseBody List<JSONResult> uploadFileHandler(
            @RequestParam("files") MultipartFile file, HttpServletRequest request) {
        logger.info("Starting upload of file: " + file.getOriginalFilename());
        JSONResult result = null;

            try {
                result = uploadFile(file, appUserDTO, result, request);
            } catch (IllegalStateException | IOException e) {
                logger.error(e.getMessage() + e.getStackTrace());
                errorLogService.saveErrorLog("FileUploadController: uploadFileHandler. Error: "+ e.getMessage(), appUserDTO.getUser().getUsrUid());
            }


        List<JSONResult> array = new ArrayList<>();
        array.add(result);
        return array;
    }

Please find below method for processing the file

    public CsvFileReader(String path, String delimeter, File file) throws FileNotFoundException {
    String line="";
    rows = new ArrayList<>();
    try (BufferedReader br = new BufferedReader(new InputStreamReader(new FileInputStream(path), "UTF8"))) {            
        while ((line = br.readLine()) != null) {

            String[] lineData = line.split(delimeter,-1);
            if(SanityCheck.isValid(lineData)){
                rows.add(lineData);
            }               
        }
    } catch (IOException e) {
        logger.error(e.getMessage());
    }
}

Any one can point me out to the right direction how to solve this please?

Answer 1

Your program tries to read the files in UTF-8, therefore the files need to be in UTF-8 and it won't work if they aren't.

If you're asking how to handle files that can be in any encoding, the encoding of a file cannot be guessed, so you need to inform the server of the file's encoding when you upload it, using extra information such as a form field indicating the encoding.

If you're asking how to handle files that can be in any encoding, while you don't know how where to obtain from the encoding of a file because the files are just stashed there and you're not aware of any listing of the encoding of each of them, well like I said, it cannot be guessed.

If you feel like it, you can attempt to guess the encoding of the file, by reading it in UTF-8 first, and checking whether the result contains invalid characters. If not, reading it in UTF-8 was most likely correct. If there are invalid characters, then it's probable UTF-8 was not the correct encoding and you should try another. That other encoding may be windows-1252... And it may be something else entirely. No way to know, really.