Why the speed of a Java process inside multiple loops slows down as it goes?

Question

I'm using Win 7, Java 8, on a multicore Intel workstation. I have something like this :

  int SampleWeeksArray[]=new int[]{23,25,33},IdArray[]=new int[]{0,1,2};

  for (int StartDay=25;StartDay<36;StartDay++)
    for (int SampleWeeks : SampleWeeksArray)
      for (int Id : IdArray)
        for (float ValueFactor=0.08f;ValueFactor<0.13f;ValueFactor+=0.01)
        {
          DataDigester dataDigester=new DataDigester();
          dataDigester.StartDay=StartDay;
          dataDigester.SampleWeeks=SampleWeeks;
          dataDigester.Id=Id;
          dataDigester.ValueFactor=ValueFactor;
          dataDigester.DoRun();
        }

Inside dataDigester.DoRun() it reads in some text data files, parse each line and do some statistics with arrays, vectors and hashmaps and it uses ThreadPoolExecutor to run multiple processes, then save the results to a file.

The strange thing is each single run of DoRun() takes only 10 minutes, but inside the loops, it started with 10 min. but gradually slows down to 20, 30 min. for each DoRun().

If I stop the loop when it's taking 30 min., then start over from where I stopped, it again took 10 min. and eventually slows down to 20, 30 min. again. Why ?

I even put in the following lines after DoRun(), but it didn't improve speed, still slows down :

   Thread.sleep(20*1000);                     // Rest for 20 seconds
   System.gc();

Inside each loop, DataDigester dataDigester=new DataDigester() creates a new object, it should be recycled after the DoRun(), why it slowed down ?

I've also noticed from Windows Task Manager, the memory usage goes up as the loops keep going. It seems Java somehow burdens itself with something and slows down, so in this case how to keep the speed at a constant of 10 min. for each run within the loops ?

Answer 1

It's hard to say without knowing what DataDigester is and does, but I can think of a number of possible reasons:

1. if DataDigester.DoRun starts up a new thread - then as the number of threads goes up, the time available per thread goes down. If it utilizes the hardware threads, then the time will go significantly up once you go from 1 to 2 threads per hardware thread.

2. DataDigester may have a static structure that it adds to, and which grows slower and slower the more you add into it

Answer 2

Declare just one DataDigester and reuse that for all of your loops instead of creating thousands and thousands of them.

int SampleWeeksArray[]=new int[]{23,25,33},IdArray[]=new int[]{0,1,2};

DataDigester dataDigester;
for (int StartDay=25;StartDay<36;StartDay++)
    for (int SampleWeeks : SampleWeeksArray)
        for (int Id : IdArray)
            for (float ValueFactor=0.08f;ValueFactor<0.13f;ValueFactor+=0.01)
            {
                dataDigester=new DataDigester();
                dataDigester.StartDay=StartDay;
                dataDigester.SampleWeeks=SampleWeeks;
                dataDigester.Id=Id;
                dataDigester.ValueFactor=ValueFactor;
                dataDigester.DoRun();
            }
        }
    }
}

That should help with your Garbage Collection issues and the timing too.