After renaming files with Java some not accessible on linux system or via Java

Question

I'm having an issue with files reading (and perhaps writing) files on a linux system using Java. My application was complaining it could not read some audio files and when I looked on the system I noticed ls -l failed on these files as well and that all the problem files were the ones containing characters with quotes etc such as é , files without these characters are okay.

[root@N1-0247 Georges Bizet- Suites from Carmen & L'arlésienne]# pwd
/mnt/disk1/share/import/all/MusicUnmatched/WAV/Yan Pascal Tortelier/Georges Bizet- Suites from Carmen & L'arlésienne
[root@N1-0247 Georges Bizet- Suites from Carmen & L'arlésienne]# ls -l
ls: cannot access 20 - L' Arlésienne, suite for orchestra No. 1, from the incidental music- Farandole.WAV: No such file or directory
ls: cannot access 19 - L' Arlésienne, suite for orchestra No. 1, from the incidental music- Minuetto.WAV: No such file or directory
ls: cannot access 18 - L' Arlésienne, suite for orchestra No. 1, from the incidental music- Intermezzo.WAV: No such file or directory
ls: cannot access 17 - L' Arlésienne, suite for orchestra No. 1, from the incidental music- Pastorale.WAV: No such file or directory
ls: cannot access 16 - L' Arlésienne, suite for orchestra No. 1, from the incidental music- Carillon.WAV: No such file or directory
ls: cannot access 15 - L' Arlésienne, suite for orchestra No. 1, from the incidental music- Adagietto.WAV: No such file or directory
ls: cannot access 14 - L' Arlésienne, suite for orchestra No. 1, from the incidental music- Minuetto.WAV: No such file or directory
ls: cannot access 13 - L' Arlésienne, suite for orchestra No. 1, from the incidental music- Prélude.WAV: No such file or directory
ls: cannot access 08 - Carmen Suites for orchestra Nos. 1 & 2 (assembled by Ernest Guirard)- Chanson Du Toréador (Act II).WAV: No such file or directory
ls: cannot access 07 - Carmen Suites for orchestra Nos. 1 & 2 (assembled by Ernest Guirard)- Dans Bohème (Gypsy Song, Act II).WAV: No such file or directory
ls: cannot access 05 - Carmen Suites for orchestra Nos. 1 & 2 (assembled by Ernest Guirard)- Seguédille (Act I).WAV: No such file or directory
ls: cannot access 04 - Carmen Suites for orchestra Nos. 1 & 2 (assembled by Ernest Guirard)- Habeñera (Act I).WAV: No such file or directory
ls: cannot access 02 - Carmen Suites for orchestra Nos. 1 & 2 (assembled by Ernest Guirard)- Prélude (Prelude To Act I).WAV: No such file or directory
ls: cannot access 01 - Carmen Suites for orchestra Nos. 1 & 2 (assembled by Ernest Guirard)- Les Toréadors (Introduction To Act I).WAV: No such file or directory
total 192148
?????????? ? ?    ?           ?            ? 01 - Carmen Suites for orchestra Nos. 1 & 2 (assembled by Ernest Guirard)- Les Toréadors (Introduction To Act I).WAV
?????????? ? ?    ?           ?            ? 02 - Carmen Suites for orchestra Nos. 1 & 2 (assembled by Ernest Guirard)- Prélude (Prelude To Act I).WAV
-rw-rw-rw- 1 root root 36681194 Feb 21  2017 03 - Carmen Suites for orchestra Nos. 1 & 2 (assembled by Ernest Guirard)- La Grade Montante (Street Urchins' Chorus, Act I).WAV
?????????? ? ?    ?           ?            ? 04 - Carmen Suites for orchestra Nos. 1 & 2 (assembled by Ernest Guirard)- Habeñera (Act I).WAV
?????????? ? ?    ?           ?            ? 05 - Carmen Suites for orchestra Nos. 1 & 2 (assembled by Ernest Guirard)- Seguédille (Act I).WAV
-rw-rw-rw- 1 root root 16455464 Feb 21  2017 06 - Carmen Suites for orchestra Nos. 1 & 2 (assembled by Ernest Guirard)- Les Dragons D'Alcala (Entr'acte, Act II).WAV
?????????? ? ?    ?           ?            ? 07 - Carmen Suites for orchestra Nos. 1 & 2 (assembled by Ernest Guirard)- Dans Bohème (Gypsy Song, Act II).WAV
?????????? ? ?    ?           ?            ? 08 - Carmen Suites for orchestra Nos. 1 & 2 (assembled by Ernest Guirard)- Chanson Du Toréador (Act II).WAV
-rw-rw-rw- 1 root root 27743402 Feb 21  2017 09 - Carmen Suites for orchestra Nos. 1 & 2 (assembled by Ernest Guirard)- Intermezzo (Entr'acte, Act III).WAV
-rw-rw-rw- 1 root root 39886886 Feb 21  2017 10 - Carmen Suites for orchestra Nos. 1 & 2 (assembled by Ernest Guirard)- Marche Des Contrebandiers (Introduction To Act III).WAV
-rw-rw-rw- 1 root root 52822606 Feb 21  2017 11 - Carmen Suites for orchestra Nos. 1 & 2 (assembled by Ernest Guirard)- Nocturne (Micaela's Aria, Act III).WAV
-rw-rw-rw- 1 root root 23100378 Feb 21  2017 12 - Carmen Suites for orchestra Nos. 1 & 2 (assembled by Ernest Guirard)- Argonaise (Entr'acte, Act IV).WAV
?????????? ? ?    ?           ?            ? 13 - L' Arlésienne, suite for orchestra No. 1, from the incidental music- Prélude.WAV
?????????? ? ?    ?           ?            ? 14 - L' Arlésienne, suite for orchestra No. 1, from the incidental music- Minuetto.WAV
?????????? ? ?    ?           ?            ? 15 - L' Arlésienne, suite for orchestra No. 1, from the incidental music- Adagietto.WAV
?????????? ? ?    ?           ?            ? 16 - L' Arlésienne, suite for orchestra No. 1, from the incidental music- Carillon.WAV
?????????? ? ?    ?           ?            ? 17 - L' Arlésienne, suite for orchestra No. 1, from the incidental music- Pastorale.WAV
?????????? ? ?    ?           ?            ? 18 - L' Arlésienne, suite for orchestra No. 1, from the incidental music- Intermezzo.WAV
?????????? ? ?    ?           ?            ? 19 - L' Arlésienne, suite for orchestra No. 1, from the incidental music- Minuetto.WAV
?????????? ? ?    ?           ?            ? 20 - L' Arlésienne, suite for orchestra No. 1, from the incidental music- Farandole.WAV

The filesystem is UTF8 I think, at least if I set

export LANG=en_US.UTF-8

in my profile filenames display with correct names.

Earlier on these files were renamed by the Java application to the new name so although error was reported it seems to be maybe some issue with the Java application, but I dont know what.

In my Java start script I have the line

export LC_ALL=en_US.UTF-8

I have had not encountered this problem on other linux systems or Windows, MacOS ecetera.

Answer 1

尝试在 java start 命令中声明-Dfile.encoding=UTF-8 。

Answer 2

First, two points :

1) The fact that you are getting errors with ls shows that the problem is an issue between the filenames and the filesystem, not Java per se. You would get the same issue whatever language your program was written in - or indeed, if you tried to copy or rename a file directly on the command line.

2) The problem is not with the quote character, as is shown by the fact that the quote character appears in files that were named correctly - eg :

-rw-rw-rw- 1 root root 52822606 Feb 21  2017 11 - Carmen Suites for orchestra Nos. 1 & 2 (assembled by Ernest Guirard)- Nocturne (Micaela's Aria, Act III).WAV

So, the problem is with the unicode character é .

This character is this one : https://www.compart.com/en/unicode/U+00E9 , so it consists of a null byte followed by E9.

The trouble is that POSIX filesystems like xfs do not allow null bytes in filenames (see What are all the illegal characters in the XFS filesystem? )

Upshot is, you cannot have filenames with THAT character in THAT filesystem.

So, you have to change either the filename, or the filesystem.

For example, this page lists filesystems, indicating those that allow unicode in their filenames :

https://en.wikipedia.org/wiki/Filename#Comparison_of_filename_limitations

(as an aside, in that list is Apple's HFS+, but interesting to note that that has been replaced by the Apple File System APFS that does NOT allow unicode in filenames - https://developer.apple.com/library/archive/documentation/FileManagement/Conceptual/APFS_Guide/FAQ/FAQ.html )

The other alternative to change your Java program to modify the filename to replace é with e :

    String safeFilename = filename.replaceAll("é", "e");

or if you prefer :

    String safeFilename = filename.replaceAll( "\u00e9", "e" );

Answer 3

That is a nice challenging puzzle you got there as this is hard to debug

I tried to reproduce with java.nio.Files with Java 11 on Debian 10 with XFS and bash (as ls builtin) but could not reproduce the issue with files named é.

Please try to get to a simpler reproduction scenario and more details otherwise I just have to guess that this may have to do with:

• A compatibility issue with Unicode normalization, see: https://docs.oracle.com/javase/tutorial/i18n/text/normalizerapi.html
• not yet using modern Java API's like java.nio.Files like in this question java.io.File: accessing files with invalid filename encodings

The problem is also that this seems to be hard to debug as simple tools like strace do not show enough information on the bytes of the filenames getdents syscall to see what is going on at the lower level API's, see: for what I mean

Maybe it is time for a different strategy? Try to only write the full title of the songs to a playlist file ? There will always be special chars that will in some setting cause problems, even spaces with scripts if you are uncareful, directory seperators (slash or backslash) etc. (See: this related question )

Answer 4

Your filesystem is corrupted - it is not an application level issue, but the content on your physical disk is not valid according to the filesystem driver that converts the physical disk content into filenames + data. You need to check which device represents your file system (the "mount" command shows which device is mounted into which directory - it is probably something like /dev/sda1 . You need to re-mount it as read-only (which can be tricky if this is your root filesystem) and run fsck /dev/sda1 (or whatever your device is) to repair it. It is not 100% sure that you can get those files back.