Approximation with Neural Network not working, predicts a constant value throughout the domain

Question

I'm writing a script to infer a plane from 2D data. So far, this is what I came up with:

import matplotlib.pyplot as plt
import numpy as np


def interp(dataxy,dataz,prediction,plot):
    model=tf.keras.Sequential([tf.keras.layers.Dense(1),
                           tf.keras.layers.Dense(10,activation=tf.nn.relu),
                            tf.keras.layers.Dense(1,activation=None)])
    model.compile(loss='mean_squared_error',optimizer=tf.keras.optimizers.Adam(0.1))
    model.fit(dataxy,dataz,epochs=100,verbose=False)

    if plot:
        iter_max=10
        x=y=np.arange(0,1+1/iter_max,1/iter_max)
        Z=np.zeros((iter_max+1,iter_max+1))

        for iter_i in x:
            for iter_j in y:

                ponto = [iter_i,iter_j]
                [aux] = model.predict([ponto])
                Z[int(iter_max*iter_i)][int(iter_max*iter_j)]=aux
                
        X,Y = np.meshgrid(x,y)
        Z=Z.reshape(iter_max+1,iter_max+1)
        
        fig = plt.figure()
        ax = fig.gca(projection='3d')
        # plt.xlabel('input')
        # plt.ylabel("output")
        ax.scatter([dataxy[i][0] for i in range(0,len(dataxy))],[dataxy[i][1] for i in range(0,len(dataxy))], zs=dataz, s=20, color='black')
        ax.plot_surface(X,Y,Z, label="predictions")
    
    return model.predict([prediction])

And the training data is given by:

xx=[[0,0],[0,1],[0,1],[1,0],[0,1],[0,0],[1,1]]
and
yy=[2.6,0.9,1.2,-0.05,1.1,2.3,-1.42]

The thing is, it's only returning a constant value, about 2.4, throughout the whole domain ([0,1]x[0,1]). What might be the problem here?

Answer 1

Solved. Here's the final code.

import tensorflow as tf
import matplotlib.pyplot as plt
import numpy as np


def interpv3(dataxy,dataz,prediction,plot):
    model=tf.keras.Sequential()
    model.add(tf.keras.layers.Dense(10,activation=tf.nn.relu, input_dim = 2))
    model.add(tf.keras.layers.Dense(1,activation=None))
    model.compile(loss='mean_squared_error',optimizer=tf.keras.optimizers.Adam(0.1))
    model.fit(dataxy,dataz,epochs=100,verbose=False)

    if plot:
        iter_max=10
        x=y=np.arange(0,1+1/iter_max,1/iter_max)
        Z=np.zeros((iter_max+1,iter_max+1))

        for iter_i in x:
            for iter_j in y:
                
                ponto = [iter_i,iter_j]
                [aux] = model.predict([ponto])
                Z[int(iter_max*iter_i)][int(iter_max*iter_j)]=aux
                
        X,Y = np.meshgrid(x,y)
        Z=Z.reshape(iter_max+1,iter_max+1)
        
        fig = plt.figure()
        ax = fig.gca(projection='3d')
        ax.scatter([dataxy[i][0] for i in range(0,len(dataxy))],
                   [dataxy[i][1] for i in range(0,len(dataxy))],
                   dataz, s=20, color='black')
        ax.plot_surface(X,Y,Z, label="predictions")
    
    return model.predict([prediction])

The difference was that I explicitly said that the input_dim was 2, instead of just adding a Dense layer with 2 neurons. Needs some tinkering with number of hidden neurons and learning_rate, though.

Answer 2

The NN failed to learn a function from your data. There are two problems:

1. As you can see in your data, two identical samples are corresponding to two different expected values.
2. The amount of data is insufficient for the model to learn.

I suggest collecting more data first, or you could do a simple data augmentation by adding a small random noise to your inputs.