具有1個隱藏層神經網絡的Tensorflow中的預測不會更改-回歸

Question

我是TensorFlow和Neural Networks的新手，我正在嘗試開發一個可以預測屬性值的神經網絡（這是在Kaggle.com上的入門比賽的一部分），我知道使用Neural Network也許不是解決回歸問題的最佳模式，但我決定試試看。

當使用單層神經網絡（沒有隱藏層，可能是線性回歸）時，模型實際上預測的值接近實際值，但是當我添加隱藏層時，所有預測的值與一批20個輸入的值相同張量

   ('real', array([[ 181000.],
       [ 128900.],
       [ 161500.],
       [ 180500.],
       [ 181000.],
       [ 183900.],
       [ 122000.],
       [ 378500.],
       [ 381000.],
       [ 144000.],
       [ 260000.],
       [ 185750.],
       [ 137000.],
       [ 177000.],
       [ 139000.],
       [ 137000.],
       [ 162000.],
       [ 197900.],
       [ 237000.],
       [  68400.]]))
('prediction ', array([[ 4995.10597687],
       [ 4995.10597687],
       [ 4995.10597687],
       [ 4995.10597687],
       [ 4995.10597687],
       [ 4995.10597687],
       [ 4995.10597687],
       [ 4995.10597687],
       [ 4995.10597687],
       [ 4995.10597687],
       [ 4995.10597687],
       [ 4995.10597687],
       [ 4995.10597687],
       [ 4995.10597687],
       [ 4995.10597687],
       [ 4995.10597687],
       [ 4995.10597687],
       [ 4995.10597687],
       [ 4995.10597687],
       [ 4995.10597687]]))

更新：我已經注意到預測的值僅反映輸出層的偏差，而隱藏層和輸出層的權重都沒有變化，並且始終為零

為了進一步檢查出問題所在，我生成了模型的圖（一次使用隱藏層時，一次不使用隱藏層）以比較兩個圖，看看是否缺少某些東西，不幸的是它們都看起來對我來說是正確的，但是我仍然不明白為什么在沒有隱藏層的情況下模型可以工作，而在使用隱藏層的情況下不能工作的模型

工作模型圖（中間沒有隱藏層）：

不工作模型的圖形（具有隱藏層和輸出層）

我的完整代碼如下：

# coding: utf-8
import tensorflow as tf 
import numpy as np 
def loadDataFromCSV(fileName , numberOfFields , numberOfOutputFields , numberOfRecords):
    XsArray = np.ndarray([numberOfRecords ,(numberOfFields-numberOfOutputFields)] , dtype=np.float64)
    YsArray = np.ndarray([numberOfRecords ,numberOfOutputFields] , dtype=np.float64)
    fileQueue = tf.train.string_input_producer(fileName)
    defaultValues = [[0]]*numberOfFields
    decodedLine = [[None]]*numberOfFields
    reader  = tf.TextLineReader()
    key , singleLine = reader.read(fileQueue)
    decodedLine = tf.decode_csv(singleLine,record_defaults=defaultValues)
    inputFeatures = decodedLine[0:numberOfFields-numberOfOutputFields]
    outputFeatures =decodedLine[numberOfFields-numberOfOutputFields:numberOfFields]
    with tf.Session() as session : 
        tf.global_variables_initializer().run()
        coor = tf.train.Coordinator()
        threads = tf.train.start_queue_runners(coord=coor)
        for i in range(numberOfRecords) :
            XsArray[i,:] ,YsArray[i,:]  = session.run([inputFeatures , outputFeatures]) 
        coor.request_stop()
        coor.join(threads)
    return XsArray , YsArray
x , y =loadDataFromCSV(['/Users/mousaalsulaimi/Downloads/convertcsv.csv'] , 289 , 1, 1460)
num_steps = 10000
batch_size = 20 

graph = tf.Graph()
with graph.as_default() :
    with tf.name_scope('input'):
        inputProperties  = tf.placeholder(tf.float32 , shape=(batch_size ,287 ))
    with tf.name_scope('realPropertyValue') :
        outputValues = tf.placeholder(tf.float32,shape=(batch_size,1))
    with tf.name_scope('weights'):
        hidden1_w  = tf.Variable( tf.truncated_normal([287,1000],stddev=math.sqrt(3/(287+1000)) , dtype=tf.float32))
    with tf.name_scope('baises'):
        hidden1_b = tf.Variable( tf.zeros([1000] , dtype=tf.float32) )
    with tf.name_scope('hidden_layer'):
        hidden1 =tf.matmul(inputProperties,hidden1_w) + hidden1_b
    #hidden1_relu = tf.nn.relu(hidden1)
    #hidden1_dropout = tf.nn.dropout(hidden1_relu,.5)
    with tf.name_scope('layer2_weights'):
        output_w  = tf.Variable(tf.truncated_normal([1000,1],stddev=math.sqrt(3/(1000+1)) , dtype=tf.float32))
    with tf.name_scope('layer2_baises'):
        output_b = tf.Variable(tf.zeros([1] , dtype=tf.float32))
    with tf.name_scope('layer_2_predictions'):
        output =tf.matmul(hidden1,output_w) + output_b
    with tf.name_scope('predictions'):
        predictedValues = (output)
    loss = tf.sqrt(tf.reduce_mean(tf.square(predictedValues-outputValues)))
    loss_l2 = tf.nn.l2_loss(hidden1_w)
    with tf.name_scope('minimization') :
        minimum = tf.train.AdamOptimizer(.5).minimize(loss+.004*loss_l2)

with tf.Session(graph=graph) as session:
    tf.global_variables_initializer().run()
    print("Initialized")
    for step in range(num_steps):
        # Pick an offset within the training data, which has been randomized.
        # Note: we could use better randomization across epochs.
        offset = (step * batch_size) % (y.shape[0] - batch_size)
        # Generate a minibatch.
        batch_data = x[offset:(offset + batch_size), 1:]
        batch_labels = y[offset:(offset + batch_size), :]
        print("real" , batch_labels)
        # Prepare a dictionary telling the session where to feed the minibatch.
        # The key of the dictionary is the placeholder node of the graph to be fed,
        # and the value is the numpy array to feed to it.
        feed_dict = {inputProperties : batch_data, outputValues : batch_labels}
        _, l, predictions  , inp  = session.run([minimum, loss, predictedValues  ,inputProperties ], feed_dict=feed_dict)
        print("prediction " , predictions)
        print("loss : " , l)
        print("----------")

        print('+++++++++++')

另外，如果您想看看，我已經在這里上傳了數據文件convertcsv.csv。

我感謝任何幫助找出我做錯了什么。

謝謝

Answer 1

好的，所以我終於明白了問題所在，並且正如預期的那樣，它是神經網絡中的權重，我還進行了一些預處理以增強預測：

import tensorflow as tf
import numpy as np
import math
from sklearn import preprocessing

def loadDataFromCSV(fileName , numberOfFields , numberOfOutputFields , numberOfRecords):
    XsArray = np.ndarray([numberOfRecords ,(numberOfFields-numberOfOutputFields)] , dtype=np.float64)
    YsArray = np.ndarray([numberOfRecords ,numberOfOutputFields] , dtype=np.float64)
    fileQueue = tf.train.string_input_producer(fileName)
    defaultValues = [[0]]*numberOfFields
    decodedLine = [[None]]*numberOfFields
    reader  = tf.TextLineReader()
    key , singleLine = reader.read(fileQueue)
    decodedLine = tf.decode_csv(singleLine,record_defaults=defaultValues)
    inputFeatures = decodedLine[0:numberOfFields-numberOfOutputFields]
    outputFeatures =decodedLine[numberOfFields-numberOfOutputFields:numberOfFields]
    with tf.Session() as session :
        tf.global_variables_initializer().run()
        coor = tf.train.Coordinator()
        threads = tf.train.start_queue_runners(coord=coor)
        for i in range(numberOfRecords) :
            XsArray[i,:] ,YsArray[i,:]  = session.run([inputFeatures , outputFeatures])
        coor.request_stop()
        coor.join(threads)
    return XsArray , YsArray
x , y =loadDataFromCSV(['/Users/mousaalsulaimi/Downloads/convertcsv.csv'] , 289 , 1, 1460)
num_steps = 10000
batch_size = 20



graph = tf.Graph()
beta = .00009
with graph.as_default() : 
     keepprop = tf.placeholder( tf.float32 , shape=([1]) )
     with tf.name_scope('input'):
         inputProperties  = tf.placeholder(tf.float32 , shape=(None ,287 ))
     with tf.name_scope('realPropertyValue') : 
         outputValues = tf.placeholder(tf.float32,shape=(None,1))
     with tf.name_scope('weights'):
         hidden1_w  = tf.Variable( tf.truncated_normal([287,2000],stddev=math.sqrt(3/(1)) , dtype=tf.float32))
     with tf.name_scope('baises'):
         hidden1_b = tf.Variable( tf.zeros([2000] , dtype=tf.float32) )
     with tf.name_scope('hidden_layer'):
         hidden1 =tf.matmul(inputProperties,hidden1_w) + hidden1_b  
         hidden1_relu = tf.nn.relu(hidden1)
         hidden1_dropout = tf.nn.dropout(hidden1_relu,keep_prob=keepprop[0])
     with tf.name_scope('layer2_weights'):
         hidden2_w  = tf.Variable(tf.truncated_normal([2000,500],stddev=math.sqrt(3/(1)) , dtype=tf.float32))
     with tf.name_scope('layer2_baises'):
         hidden2_b = tf.Variable(tf.zeros([500] , dtype=tf.float32))
     with tf.name_scope('layer_2'):
         hidden2 =tf.matmul(hidden1_dropout,hidden2_w) + hidden2_b
         hidden2_relu = tf.nn.relu(hidden2)
     hidden2_dropout= tf.nn.dropout(hidden2_relu,keepprop[0])
     with tf.name_scope('output_layer_weights'): 
         output_w = tf.Variable(tf.truncated_normal([500,1],stddev=math.sqrt(3/(1)) , dtype=tf.float32))
     with tf.name_scope('outout_layer_baises'):
         output_b = tf.Variable(tf.zeros([1] , dtype=tf.float32))
     with tf.name_scope('output_layer'):
         output = tf.matmul(hidden2_dropout,output_w) + output_b    
     with tf.name_scope('predictions'):
         predictedValues = tf.nn.relu(output)
     loss = tf.sqrt(tf.reduce_mean(tf.square((predictedValues)-(outputValues))))
     loss_l2 = tf.nn.l2_loss(hidden1_w) + tf.nn.l2_loss(hidden2_w) + tf.nn.l2_loss(output_w) + tf.reduce_mean(output_w) + tf.reduce_mean(hidden2_w) + tf.reduce_mean(hidden1_w)
     global_step = tf.Variable(0,trainable=False)
     start_step = .5 
     learning_rate = tf.train.exponential_decay(start_step ,global_step , 100 , .94 , staircase=True)
     with tf.name_scope('minimization') : 
         minimum = tf.train.AdadeltaOptimizer(learning_rate).minimize(loss+beta*loss_l2 , global_step=global_step)

with tf.Session(graph=graph) as session:
    tf.global_variables_initializer().run()
    '''writer = tf.summary.FileWriter('/Users/mousaalsulaimi/Downloads/21' , graph=graph)'''
    num_steps = 1000
    batch_size = 730
    print("Initialized")
    for step in range(num_steps):
        # Pick an offset within the training data, which has been randomized.
        # Note: we could use better randomization across epochs.
        offset = (step * batch_size) % (y.shape[0] - batch_size)
            # Generate a minibatch.

        batch_data_ss = preprocessing.MinMaxScaler().fit(x[offset:(offset + batch_size), 1:])
        batch_data = batch_data_ss.transform(x[offset:(offset + batch_size), 1:])
        batch_labels = y[offset:(offset + batch_size), :]
        # Prepare a dictionary telling the session where to feed the minibatch.
        # The key of the dictionary is the placeholder node of the graph to be fed,
        # and the value is the numpy array to feed to it.
        feed_dict = {keepprop:[.65], inputProperties : batch_data, outputValues : batch_labels }
        _, l, predictions  , inp , w_l   = session.run([minimum, loss, predictedValues  ,inputProperties , hidden1_w   ], feed_dict=feed_dict)
        print("loss2 : " , l )
        print("loss : " , accuricy((predictions) ,( batch_labels)) )

以下是來自預測的樣本

('loss : ', 0.15377927727091956)
('loss2 : ', 29109.197)
('loss : ', 0.1523804301893735)
('loss2 : ', 29114.414)
('loss : ', 0.15479254974665729)
('loss2 : ', 30617.834)
('loss : ', 0.15270011182205656)
('loss2 : ', 29519.598)
('loss : ', 0.15641723449772593)
('loss2 : ', 29307.811)
('loss : ', 0.15460120852074882)
('loss2 : ', 27985.998)
('loss : ', 0.14993038617463786)
('loss2 : ', 28811.738)
('loss : ', 0.1549284462882819)
('loss2 : ', 29157.725)
('loss : ', 0.15402833737387819)
('loss2 : ', 27079.215)
('loss : ', 0.14974744509723023)
('loss2 : ', 26622.93)
('loss : ', 0.1419577502544874

這些預測並不完美，但有所進展，正如您所看到的，每個房產的預測都減少了30,000美元。