向Tensorflow添加自定义正则化

Question

I am using tensorflow to optimize a simple least squares objective function like the following:

Here, Y is the target vector , X is the input matrix and vector w represents the weights to be learned.

Example Scenario:

, ,

If I wanted to augment the initial objective function to impose an additional constraint on w1 (the first scalar value in the tensorflow Variable w and X1 represents the first column of the feature matrix X ), how would I achieve this in tensorflow?

One solution I can think of is to use tf.slice to index the first value of $w$ and add this in addition to the original cost term but I am not convinced that it will have the desired effect on the weights.

I would appreciate inputs on whether something like this is possible in tensorflow and if so, what the best ways to implement this might be?

An alternate option would be to add weight constraints, and do it using an augmented Lagrangian objective but I would first like to explore the regularization option before going the Lagrangian route.

The current code I have for the initial objective function without additional regularization is the following:

train_x ,train_y are the training data, training targets respectively.
test_x  , test_y are the testing data, testing targets respectively.

#Sum of Squared Errs. Cost.
def costfunc(predicted,actual):
    return tf.reduce_sum(tf.square(predicted - actual))

#Mean Squared Error Calc.
def prediction(sess,X,y_,test_x,test_y):
    pred_y = sess.run(y_,feed_dict={X:test_x})
    mymse = tf.reduce_mean(tf.square(pred_y - test_y))
    mseval=sess.run(mymse)

    return mseval,pred_y


with tf.Session() as sess:

    X = tf.placeholder(tf.float32,[None,num_feat])  #Training Data 
    Y = tf.placeholder(tf.float32,[None,1]) #  Target Values 
    W = tf.Variable(tf.ones([num_feat,1]),name="weights")  

    init = tf.global_variables_initializer()

    sess.run(init)

    #Tensorflow ops and cost function definitions.
    y_ = tf.matmul(X,W) 
    cost_history = np.empty(shape=[1],dtype=float)
    out_of_sample_cost_history = np.empty(shape=[1],dtype=float)
    cost=costfunc(y_,Y) 
    learning_rate = 0.000001
    training_step = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)


    for epoch in range(training_epochs):
        sess.run(training_step,feed_dict={X:train_x,Y:train_y})
        cost_history = np.append(cost_history,sess.run(cost,feed_dict={X: train_x,Y: train_y}))
        out_of_sample_cost_history = np.append(out_of_sample_cost_history,sess.run(cost,feed_dict={X:test_x,Y:test_y}))


    MSETest,pred_test = prediction(sess,X,y_,test_x,test_y) #Predict on full testing set.

Answer 1

tf.slice will do. And during optimization, the gradients to w1 will be added (because gradients add up at forks). Also, please check the graph on Tensorboard (the link on how to use it).