梯度下降的θ值是什么意思？

Question

I have all the components, I just am not quite sure This is my output:

Theta-->: 0.09604203456288299, 1.1864676227195392

How do I interpret that? What does it mean?

I essentially just modified the example from this description . But I'm not sure if it's really applicable to my problem. I'm trying to perform binary classification on a set of documents. The documents are rendered as bag-of-words style feature vectors of the form:

Example:

Document 1 = ["I", "am", "awesome"]
Document 2 = ["I", "am", "great", "great"]

Dictionary is:

["I", "am", "awesome", "great"]

So the documents as a vector would look like:

Document 1 = [1, 1, 1, 0]
Document 2 = [1, 1, 0, 2]

This is my gradient descent code:

public static double [] gradientDescent(final double [] theta_in, final double alpha, final int num_iters, double[][] data ) 
{
    final double m = data.length;   
    double [] theta = theta_in;
    double theta0 = 0;
    double theta1 = 0;
    for (int i = 0; i < num_iters; i++) 
    {                        
        final double sum0 = gradientDescentSumScalar0(theta, alpha, data );
        final double sum1 = gradientDescentSumScalar1(theta, alpha, data);                                   
        theta0 = theta[0] - ( (alpha / m) * sum0 ); 
        theta1 = theta[1] - ( (alpha / m) * sum1 );                        
        theta = new double [] { theta0, theta1 };
    }
    return theta;
}


//data is the feature vector
//this theta is weight
protected static double [] matrixMultipleHthetaByX( final double [] theta, double[][] data ) 
{
    final double [] vector = new double[ data.length ];
    int i = 0;                 
    for (final double [] d : data) 
    {
        vector[i] = (1.0 * theta[0]) + (d[0] * theta[1]);            
        i++;
    } // End of the for // 
    return vector;
}


protected static double gradientDescentSumScalar0(final double [] theta, final double alpha, double[][] data ) 
{        
    double sum = 0;
    int i = 0;
    final double [] hthetaByXArr = matrixMultipleHthetaByX(theta, data ); 
    for (final double [] d : data) 
    {
        final double X = 1.0;
        final double y = d[1];
        final double hthetaByX = hthetaByXArr[i];    
        sum = sum + ( (hthetaByX - y) * X );
        i++;
    } // End of the for //
    return sum;
}
protected static double gradientDescentSumScalar1(final double [] theta, final double alpha, double[][] data ) 
{        
    double sum = 0;
    int i = 0;
    final double [] hthetaByXArr = matrixMultipleHthetaByX(theta, data );
    for (final double [] d : data) 
    {
        final double X = d[0];
        final double y = d[1];            
        final double hthetaByX = hthetaByXArr[i];         
        sum = sum + ( (hthetaByX - y) * X );
        i++;
    } // End of the for //
    return sum;
}

public static double [] batchGradientDescent( double [] weights, double[][] data ) 
{
    /*
     * From tex:
     * \theta_j := \theta_j - \alpha\frac{1}{m} \sum_{i=1}^m ( h_\theta (x^{(i)})
     */
    final double [] theta_in = weights;
    double [] theta = gradientDescent(theta_in, alpha, iterations, data );
    lastTheta = theta;
    System.out.println("Theta-->: " + theta[0] + ", " + theta[1]);
    return theta;
}

I call it like this:

   final int globoDictSize = globoDict.size(); // number of features

   double[] weights = new double[globoDictSize + 1];
   for (int i = 0; i < weights.length; i++) 
   {
       //weights[i] = Math.floor(Math.random() * 10000) / 10000;
       //weights[i] = randomNumber(0,1);
       weights[i] = 0.0;
   }


   int inputSize = trainingPerceptronInput.size();
   double[] outputs = new double[inputSize];
   final double[][] a = Prcptrn_InitOutpt.initializeOutput(trainingPerceptronInput, globoDictSize, outputs, LABEL);



       for (int p = 0; p < inputSize; p++) 
       {

           Gradient_Descent.batchGradientDescent( weights, a );
       }

How can I verify that this code is doing what I want? Shouldn't it be outputting a predicted label or something? I've heard I can also apply to it an error function, such as hinge loss , that would come after the call to batch gradient descent as a seperate component, isn't it?

Answer 1

You code is complicated (I used to implement batch gradient descent in Octave, not in OO programming languages). But as far as I see in your code (and it is a common to use this notation) Theta is a parameter vector. After grad descend algorithm converges it returns you optimal Theta vector. After that you could claculate output of your new example with formula:

theta_transposed * X,

where theta_trasponsed is a transposed vector of theta, X is a vector of input features.

On a side note, the example you have referred to is a regression task (it is about linear regression). While the task you describe is a classification problem, where instead of predicting some value (some number - weight, length, smth else) you need to assign a label to input set. It can be completed with lots of different algorithms, but defenetily not with linear regression which is described in article you posted.

I also need to mentioned that it is absolutely not clear what kind of classification you try to perform. In your exmaple you have a bag of words description (matrixes of word counts). But where are classificaiton labels? Is it multi-output classification? Or just multi-class? Or binary?

I really suggest you to take a course on ml. Maybe on coursera. This one is good: https://www.coursera.org/course/ml It also covers full implementaion of gradient descent.