Minimization of number of itererations by adjusting input parameters in Java

Question

I was inspired by this question XOR Neural Network in Java Briefly, a XOR neural network is trained and the number of iterations required to complete the training depends on seven parameters (alpha, gamma3_min_cutoff, gamma3_max_cutoff, gamma4_min_cutoff, gamma4_max_cutoff, gamma4_min_cutoff, gamma4_max_cutoff). I would like to minimize number of iterations required for training by tweaking these parameters.

So, I want to rewrite program from

private static double alpha=0.1, g3min=0.2, g3max=0.8; 
int iteration= 0;  
loop {
    do_something;  
    iteration++; 
    if (error < threshold){break}
    }
System.out.println( "iterations: " + iteration) 

to

for (double alpha = 0.01; alpha < 10; alpha+=0.01){
    for (double g3min = 0.01; g3min < 0.4; g3min += 0.01){   
        //Add five more loops to optimize other parameters  
        int iteration = 1; 
        loop {
            do_something;  
            iteration++; 
            if (error < threshold){break}
            }
        System.out.println( inputs ); 
        //number of iterations, alpha, cutoffs,etc  
            //Close five more loops here
        }
    } 

But this brute forcing method is not going to be efficient. Given 7 parameters and hundreds of iterations for each calculation even with 10 points for each parameter translates in billions of operations. Nonlinear fit should do, but those typically require partial derivatives which I wouldn't have in this case.

Is there a Java package for this sort of optimizations?

Thank you in advance, Stepan

Answer 1

You have some alternatives - depending on the equations that govern the error parameter.

1. Pick a point in parameter space and use an iterative process to walk towards a minimum. Essentially, add a delta to each parameter and pick whichever reduces the error by the most - rince - repeat.
2. Pick each pareameter and perform a binary-chop search between its limits to find it's minimum. Will only work if the parameter's effect is linear.
3. Solve the system using some form of Operations-Research technique to track down a minimum.