Java不会遍历大目录中的所有文件

Question

I'm doing some data mining for the first time using the enron email dataset. I'm trying to iterate through every file in a directory and parse into a csv file the date, time and addressor of every file.

The problem is that java doesn't seem to iterate through all of them, which is why my csv file is around 1000 lines too short. How can I solve this?

My code:

public class FileReader {


    public static void main(String[] args) throws FileNotFoundException{
    FileReader fileReader = new FileReader();

    //fileReader.mainFunction("maildir/skilling-j/_sent_mail");
    fileReader.mainFunction("maildir/skilling-j/inbox");
    /*fileReader.mainFunction("maildir/skilling-j/sent");
    fileReader.mainFunction("maildir/lay-k/inbox");
    fileReader.mainFunction("maildir/lay-k/_sent");
    fileReader.mainFunction("maildir/lay-k/sent");*/
    System.out.println("done!");

    }
    public void mainFunction(String fileName) throws FileNotFoundException{
    File maindir = new File(fileName);
    PrintWriter pw = new PrintWriter(new File("Analysis.csv"));
    StringBuilder sb = new StringBuilder();
    StringBuilder sbpre = new StringBuilder();

    Scanner scanner;
    sbpre.append("Date");
    sbpre.append(',');
    sbpre.append("Time");
    sbpre.append(",");
    sbpre.append("From");
    sbpre.append('\n');
    int endcounter = 0;
    pw.write(sbpre.toString());
    File [] files = maindir.listFiles();
        for(int i = 0; i < files.length; i++){
            scanner = new Scanner(files[i]);
            System.out.println(files[i].getPath());
            while (scanner.hasNextLine()) {
                String lineFromFile = scanner.nextLine();
                String month = "Jun";
                String year = "2000";
                String time = "00:00:00";
                if(lineFromFile.contains("Date:") & (lineFromFile.length()== 43 | lineFromFile.length()== 42 )){
                    if(lineFromFile.length()==43){
                        sb.append(lineFromFile.substring(11,13));
                        month = lineFromFile.substring(14, 17); 
                        year = lineFromFile.substring(18,22);
                        time = lineFromFile.substring(23,30);
                    }else{
                        sb.append("0");
                        sb.append(lineFromFile.substring(11,12)); 
                        month = lineFromFile.substring(13, 16);
                        year = lineFromFile.substring(17,21);   
                        time = lineFromFile.substring(22,29);
                                            }                   
                    sb.append(".");

                    switch(month){
                    case "Jan":sb.append("01"); sb.append(".");break;
                    case "Feb":sb.append("02"); sb.append(".");break;
                    case "Mar":sb.append("03"); sb.append(".");break;
                    case "Apr":sb.append("04"); sb.append(".");break;
                    case "May":sb.append("05"); sb.append(".");break;
                    case "Jun":sb.append("06"); sb.append(".");break;
                    case "Jul":sb.append("07"); sb.append(".");break;
                    case "Aug":sb.append("08"); sb.append(".");break;
                    case "Sep":sb.append("09"); sb.append(".");break;
                    case "Oct":sb.append("10"); sb.append(".");break;
                    case "Nov":sb.append("11"); sb.append(".");break;
                    case "Dec":sb.append("12"); sb.append(".");break;
                    }
                    sb.append(year);
                    sb.append(",");
                    sb.append(time);
                    sb.append(",");


            }

                if(lineFromFile.contains("X-From:")) {
                        lineFromFile = lineFromFile.replace(",", " ");
                        sb.append(lineFromFile.substring(8));

                    }

            pw.write(sb.toString());
            sb.setLength(0);
        }
            sb.append('\n');
            endcounter = i;
    }
        pw.close();
        System.out.println(endcounter);
    }
}

Console log last lines:

maildir\skilling-j\inbox\997_
maildir\skilling-j\inbox\998_
maildir\skilling-j\inbox\999_
maildir\skilling-j\inbox\99_
maildir\skilling-j\inbox\9_
1251
done!

It should be actually around 2500 lines.

Also would be nice to know how I can iterate through a directory with directories (eg "maildir/skilling-j") instead of a single directory with files.

And I know that the code is kind of bloated but that's the result of an incompetent coder (me).

Answer 1

listFiles() method returns list of files and directories. You could use methods isFile(), isDirectory() to identify type of file. Try this simple code to verify files in your folder:

    File[] files = maindir.listFiles();
    System.out.println("Files count: " + files.length);
    for (int i = 0; i < files.length; i++) {
        System.out.print(files[i].getAbsolutePath());
        if (files[i].isDirectory()) {
            System.out.println(" dir");
        } else if (files[i].isFile()) {
            System.out.println(" file");
        }
    }

You could use isDirectory() method to filter only directories and iterate throw them.