CSV File encoding on e-mail Intent

Question

I have a function that performs two tasks:

• Create a CSV file on data/files
• Attach the file on e-mail

The data is received raw from a REST API. Here is my function:

 private void writeCSV(String data, int year) {
    try {
        OutputStreamWriter outputStreamWriter = new OutputStreamWriter(getActivity().openFileOutput(year+"_Report.csv",
     Context.MODE_PRIVATE), StandardCharsets.UTF_8);
        outputStreamWriter.write(data);
        outputStreamWriter.close();

        File file=new File(requireActivity().getFilesDir(),year+"_Report.csv");

        Intent shareIntent = new Intent(Intent.ACTION_SEND);

        shareIntent.putExtra(Intent.EXTRA_SUBJECT, "Title"));

        Uri uri = FileProvider.getUriForFile(requireActivity(),"FileProvider",
                file);
        shareIntent.setDataAndType(uri,"application/vnd.ms-excel");
        shareIntent.putExtra(Intent.EXTRA_STREAM, uri);
        shareIntent.addFlags(Intent.FLAG_GRANT_READ_URI_PERMISSION);
        shareIntent.addFlags(Intent.FLAG_GRANT_PERSISTABLE_URI_PERMISSION);
        shareIntent.addFlags(Intent.FLAG_ACTIVITY_NEW_TASK);

        Intent chooser = Intent.createChooser(shareIntent, "Choose mail"));
        List<ResolveInfo> resInfoList = getActivity().getPackageManager().queryIntentActivities(chooser, PackageManager.MATCH_DEFAULT_ONLY);
        for (ResolveInfo resolveInfo : resInfoList) {
            String packageName = resolveInfo.activityInfo.packageName;
            getActivity().grantUriPermission(packageName, uri, Intent.FLAG_GRANT_WRITE_URI_PERMISSION | Intent.FLAG_GRANT_READ_URI_PERMISSION);
        }

        startActivity(chooser);

    } catch (Exception e) {
        e.printStackTrace();
        Log.e(TAG, e.getMessage());
    } 
}

The Problem

The problem I have is that the CSV is readable on Device File Explorer, and on E-Mail preview (ie Gmail), but when I try to open the file on Excel all the text is not the correct encoding (Non Latin words are just symbols).

Thank you in advance

Answer 1

text/csv does not really have an exact specification and thus can use a variety of character encodings.

While some programs can be helped by using text Byte Order Marks (BOM) as the first characters to help with the encoding in some instances, but not all programs support this especially for UTF8 because it should not really need a BOM.

So most programs assume the default character encoding used by the system but Excel and other programs ask the user to specify the character encoding when opening a text/csv file.

Eg in step 1 you asked the "File origin" to select the character set.

https://support.microsoft.com/en-gb/office/text-import-wizard-c5b02af6-fda1-4440-899f-f78bafe41857

I've found that Android programs mostly assume or work out UTF8 encoding for the file as that is the default, But I've found that Excel can default to UTF16 quite often and that leads it to reading 2 character as 1 incorrectly which tends to be high up in the UTF16 character set and the result looks like symbols.

So really it is up to the user to select the right character set on import or adding the UTF8 BOM of 0xEF,0xBB,0xBF at the beginning of the csv file works for Excel from memory.