Why is my neural network having trouble predicting the next value of a sin wave?

Question

Why is my neural.network not able to predict the next number for a sin wave?

I don't know if I need a better loss function or what the issue is. It seems to optimize for about 500 steps and then it just flounders with predictions that look nothing like a wave.

The input to the model is 120. That's 120 of the previous numbers of the sin wave. The model is asked to predict where the next number will be and I store the sin wave values in a deque. I insert the next sin wave value to the end of its deque.

The targets and the predictions are arrays in a shape of (200,).
Every index in the array below 100 represents a negative number and those above 100 represent a positive number.
All values to the hundredth between -1 and 0 are represented by indexes 0 to 100.
Above 100, every hundredth between 0 to 1 are represented. In other words, 200 possible values in array form to show the neural.network what to target and what the prediction is.

The code (which I had broken up in a Jupyter Notebook)...

%matplotlib inline

import torch
import random
import numpy as np
from collections import deque
from matplotlib import pyplot as plt
from matplotlib.pyplot import figure
from torch import nn
import time
import math

plt.rcParams["figure.figsize"]=(12, 8)  

class Network(nn.Module):
    def __init__(self):
        super().__init__()
        
        self.net = nn.Sequential(
            nn.Linear(120, 136),
            nn.ReLU(),
            nn.Dropout(0.05),
            nn.Linear(136, 146),
            nn.ReLU(),
            nn.Dropout(0.05),
            nn.Linear(146, 156),
            nn.ReLU(),
            nn.Dropout(0.05),
            nn.Linear(156, 170),
            nn.ReLU(),
            nn.Linear(170, 188),
            nn.Dropout(0.05),
            nn.ReLU(),
            nn.Linear(188, 200))  

    def forward(self, x):
        return self.net(x)  

online_net = Network().cuda("cuda:2")

optimizer = torch.optim.Adam(online_net.parameters(), lr=1e-5)
device = torch.device("cuda:2" if torch.cuda.is_available() else "cpu")  

sensor_buffer = deque(maxlen=120)
action_buffer = deque(maxlen=120)

[sensor_buffer.append(np.array([random.random() for _ in range(4)]).mean()) for __ in range(121)]
[action_buffer.append(0) for _ in range(121)]  

def format_target(n):
    n = (n * 100)
    l = [-1 for o in range(200)]
    l[int(n)] = 1
    return l  

for steps in range(1400):
    
    adder += (random.random() * 0.2)
    sensor_buffer.append(adder)
    sensor_sinwave = np.sin(sensor_buffer[-1])

    outers = torch.tensor(np.sin(sensor_buffer), dtype=torch.float32).cuda("cuda:2").T

    outer = online_net.forward(outers)

    prediction = torch.argmax(outer).item()

    n_prediction = (prediction * 0.01) -1

    action_buffer.append(n_prediction)

    target = format_target(sensor_sinwave)
    target_t = torch.as_tensor(target, dtype=torch.float32, device=device)

    loss = nn.functional.smooth_l1_loss(outer, target_t)


    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    if steps % 150 == 0:
        print(loss.item(), n_prediction, sensor_sinwave)

The sin wave I'm trying to predict

My predictions

Answer 1

Your model has two main problems: one is mechanical and one is theoretical.

1. Your loss (and accompanying architecture) do not model the problem appropriately. You are effectively attempting to train a regression problem (R^n -> R), but you're framing it as a classification problem (R^n -> Z^m). Typically (and coarsely), for classification problems you'll use cross-entropy loss while for regression you'll use L2 loss. If you change the output dimension of your model to 1 and the loss to L2, you should be in better shape.

2. Your model is too complex for such a simple function. The more parameters your model has, the more variance you are likely to introduce. This amounts to a higher likelihood of overfitting and less chance of generalization. Make your model a few layers shallower and you'll likely be happier with its performance.

Answer 2

Here is a tiny C++ code to predict the next sin() value. Hopefully it is useful to you.

#include <cmath>
#include <iostream>
#include <vector>
using namespace std;

const float pi = 4.0f * atanf(1.0f);

int main(void)
{
    // Replace these samples with your input data
    vector<float> samples(120, 0.0f);

    for (size_t i = 0; i < samples.size(); i++)
    {
        const float n = (i + 1) / static_cast<float>(samples.size());
        samples[i] = sin(2 * pi * n);

        cout << samples[i] << endl;
    }

    cout << endl;

    vector<float> arc_samples(120, 0.0f);

    for (size_t i = 0; i < arc_samples.size(); i++)
    {
        arc_samples[i] = asin(samples[i]);

        cout << arc_samples[i] << endl;
    }

    cout << endl;

    size_t avg_count = 0;
    float avg_diff = 0.0f;

    for (size_t i = 1; i < arc_samples.size(); i++)
    {
        avg_diff += fabsf(arc_samples[i] - arc_samples[i - 1]);
        avg_count++;
    }

    avg_diff /= avg_count;

    float next_sample = arc_samples[arc_samples.size() - 1] + avg_diff;

    cout << "Next sin() value = " << sin(next_sample) << endl;

    return 0;
}