Keras ValueError: Dimensions must be equal, but are 6 and 9 for '{{node Equal}}

Question

This is the error

ValueError: Dimensions must be equal, but are 6 and 9 for '{{node Equal}} = Equal[T=DT_FLOAT, incompatible_shape_error=true](IteratorGetNext:1, Cast_1)' with input shapes: [?,6], [?,9]

I'm trying to give a simple Keras network a group of 9 by 3 numpy arrays of integers with an intended output of a softmax on 6 categories, with a target being a one hot categorization on 6 categories. I'm using padding to create consistent 9,3 arrays (which I'd love to be rid of but that creates a slew of other errors). Now, model.fit accepts the input and target but runs into this error during runtime. I believe it's trying to correlate each row of the 9 by 3 input array to each inner element of the target array. This is not the intended correlation I want to make. I want to take the entire 9 by 3 array and correlate it to one of the 6 categories. Clearly, I am doing something wrong but I don't know what it is. I looked around for someone with a similar issue but I couldn't find one with the {{node Equal}} portion. Most other people who had the same ValueError had a different part there that made it seem unrelated including a few that were straight-up bugs in Kera.

Here is some simple code to recreate the error.

import tensorflow as tf
import numpy as np
from tensorflow import keras

model = keras.Sequential()
model.add(keras.layers.Dense(16, input_shape=(9, 3), activation='relu'))
model.add(keras.layers.Dense(32, activation='relu'))
model.add(keras.layers.Dense(32, activation='relu'))
model.add(keras.layers.Dense(16, activation='relu'))
model.add(keras.layers.Dense(6, activation='softmax'))
model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])

example_array = np.array([
                np.array([[ 5,  0,  1],
                [ 2,  0,  1],
                [ 4,  0,  1],
                [-1, -1, -1],
                [-1, -1, -1],
                [-1, -1, -1],
                [-1, -1, -1],
                [-1, -1, -1],
                [-1, -1, -1]]), 
                np.array([[ 4,  3,  0],
                [ 4,  2,  2],
                [-1, -1, -1],
                [-1, -1, -1],
                [-1, -1, -1],
                [-1, -1, -1],
                [-1, -1, -1],
                [-1, -1, -1],
                [-1, -1, -1]]), 
                np.array([[ 3,  0,  2],
                [ 1,  1,  1],
                [ 3,  2,  0],
                [ 3,  0,  3],
                [ 1,  0,  2],
                [ 4,  1,  1],
                [ 1,  1,  1],
                [ 3,  1,  1],
                [-1, -1, -1]])])
example_target = np.array([[1, 0, 0, 0, 0, 0,],
                [0, 0, 0, 0, 1, 0,],
                [0, 0, 0, 0, 0, 1,]])

model.fit(example_array, example_target, epochs=1)

Although this seems to create an error that is slightly different with the (Cast_2, Cast_3) bit differing from the original (IteratorGetNext:1, Cast_1)

ValueError: Dimensions must be equal, but are 6 and 9 for '{{node Equal}} = Equal[T=DT_FLOAT, incompatible_shape_error=true](Cast_2, Cast_3)' with input shapes: [?,6], [?,9].

This shouldn't happen considering I took this example from a sample run of my main code but here is my main code if you wish to interact with it.

Network.py

import gym
import random
import numpy as np
import tensorflow as tf
from tensorflow import keras
from statistics import median, mean
from collections import Counter
import Mastermind

INITIAL_GAMES = 1000
#The number of avaliable colours
COLOURS = 6
GUESS_LENGTH = 4
#The number of guesses avaliable to the player
GUESSES = 10
env = Mastermind.MastermindEnv(GUESS_LENGTH, COLOURS, GUESSES)
colours = env.colours

def initial_games():
    # [OBS, MOVES]
    training_data = []
    # all scores:
    scores = []
    # just the scores that met our threshold:
    accepted_scores = []
    # iterate through however many games we want:
    for _ in range(INITIAL_GAMES):
        env.reset()
        score = 0
        # guesses and results specifically from this environment:
        game_memory = []
        # Each is given the number of guesses 
        # long plus one to garuentee no interrupts
        for t in range(GUESSES+1):
            # choose random guess
            guess = ""
            for _ in range(GUESS_LENGTH):
                guess += random.choice(colours)

            #Check guess
            observation, reward, done, info = env.step(guess)

            score += reward
            if done:
                #Memory is saved after game's completion
                game_memory = observation
                break

        # If our score is positive it means that the answer was
        # correctly guessed at which point we want to save the score
        # and the game memory
        if 10 > score > 0:
            accepted_scores.append(score)
            training_data.append(game_memory)

        # reset env to play again
        env.reset()
        # save overall scores
        scores.append(score)

    # just in case you wanted to reference later
    training_data_save = np.array(training_data)
    np.save('saved.npy',training_data_save)

    #some statistics stuff
    print('Average accepted score:',mean(accepted_scores))
    print('total games won:', len(accepted_scores))
    print(Counter(accepted_scores))

    return training_data

model = keras.Sequential()
model.add(keras.layers.Dense(16, input_shape=(9, 3), activation='relu'))
model.add(keras.layers.Dense(32, activation='relu'))
model.add(keras.layers.Dense(32, activation='relu'))
model.add(keras.layers.Dense(16, activation='relu'))
model.add(keras.layers.Dense(COLOURS, activation='softmax'))
model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])

def initial_train_model():
    training_data = initial_games()
    for index in range(GUESS_LENGTH):
        training = []
        targets = []
        for observation in training_data:
            obs = []
            for i in range(len(observation)):
                if i == len(observation)-1:
                    targets.append(ord(observation[i][0][index])-97)
                else:
                    obs.append([ord(observation[i][0][index])-97,
                    ord(observation[i][1])-48,
                    ord(observation[i][2])-48])
                i += 1
            j = 10-len(observation)
            while j > 0:
                obs.append([-1, -1, -1])
                j -= 1
            training.append(np.array(obs))
        print(training)
        training = np.array(training)
        print(keras.utils.to_categorical(targets, num_classes=COLOURS))
        one_hot_targets = np.array(keras.utils.to_categorical(targets, num_classes=COLOURS))
        #print(one_hot_targets)
        model.fit(training, one_hot_targets, epochs=1)

initial_train_model()

and

Mastermind.py

import gym
from gym.utils import seeding
import numpy as np
import random

class MastermindEnv(gym.Env):
    """
    Description:
        A code guessing board game where a series of letters must be guessed 
        which gives small pieces of information to the player.

    Observation: 
        Type: List of Lists
        Guess   Blacks  Whites
        String  int     int

    Actions:
        Type: Discrete(colours^guess_length)
        String guess
        A string of length guess_length where each character can be any colour.

    Reward:
        Reward is 10 on game completion with 1 additional
        for each remaining guess.

    Starting State:
        An empty board with a target created

    Episode Termination:
        Remaining guesses reduced to 0
        guess == target
    Solved Requirements:
        100% winrate over 20 games
    """
    """
    metadata = {
        'render.modes': ['human', 'rgb_array'],
        'video.frames_per_second' : 50
    }
    """
    def __init__(self, guess_length=4, colours=6, guesses=10):
        self.guess_length = guess_length
        self.total_guesses = guesses
        self.guesses = guesses
        self.colours = []
        for _ in range(colours):
            self.colours.append(chr(_ + 97))

        self.seed()
        self.state = []

    def seed(self, seed=None):
        self.np_random, seed = seeding.np_random(seed)
        return [seed]

    def step(self, guess):
        done = False
        correct = False
        self.guesses -= 1
        if guess == self.target:
            done = True
            correct = True

        blacks = 0
        whites = 0
        accounted = []
        for i in range(len(guess)):
            if guess[i] == self.target[i]:
                blacks += 1
                accounted.append(i)
                continue
            for j in range(len(self.target)):
                if i != j and j not in accounted:
                    if guess[i] == self.target[j]:
                        whites += 1
                        accounted.append(j)
                        break  
        self.state.append([guess, blacks, whites])

        if self.guesses == 0:
            done = True

        if not done:
            reward = 0.0
        else:
            if correct:
                reward = float(self.guesses+1)
            else:
                reward = 0.0

        return np.array(self.state), reward, done, {}

    def reset(self):
        self.state = []
        self.guesses = self.total_guesses
        # Creating a target
        target = ""
        for _ in range(self.guess_length):
            target += random.choice(self.colours)
        #print(target)
        self.target = target
        return np.array(self.state)

    def render(self, mode='human'):
        print(self.state[-1])

    def close(self):
        self.state = []
        self.guesses = self.total_guesses

Mainly, I want each (9, 3) array being correlated to a single row of the target arrays if possible.

Thank you.

Answer 1

The problem might be in the definition of your model. Your input data has too many dimensions (4 dimensions) to be fitted directly into a Dense layer (1 Dimension at the input, 1 Dimension at the output). You should add a Flatten layer before your first Dense layer. You don't need any more Flatten layers in your case as the output of a Dense layer has the same dimensions as their inputs.

Besided this, you should also define the input_shape of your model as input_shape=(9, 3, 3) as your model is composed by 3 arrays with 9x3 size each one.

Thus, your sample code should something like this:

import tensorflow as tf
import numpy as np
from tensorflow import keras

model = keras.Sequential()
model.add(tf.keras.layers.Flatten())
model.add(keras.layers.Dense(16, input_shape=(9, 3, 3), activation='relu'))
model.add(keras.layers.Dense(32, activation='relu'))
model.add(keras.layers.Dense(32, activation='relu'))
model.add(keras.layers.Dense(16, activation='relu'))
model.add(keras.layers.Dense(6, activation='softmax'))
model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])

example_array = np.array([
                np.array([[ 5,  0,  1],
                [ 2,  0,  1],
                [ 4,  0,  1],
                [-1, -1, -1],
                [-1, -1, -1],
                [-1, -1, -1],
                [-1, -1, -1],
                [-1, -1, -1],
                [-1, -1, -1]]), 
                np.array([[ 4,  3,  0],
                [ 4,  2,  2],
                [-1, -1, -1],
                [-1, -1, -1],
                [-1, -1, -1],
                [-1, -1, -1],
                [-1, -1, -1],
                [-1, -1, -1],
                [-1, -1, -1]]), 
                np.array([[ 3,  0,  2],
                [ 1,  1,  1],
                [ 3,  2,  0],
                [ 3,  0,  3],
                [ 1,  0,  2],
                [ 4,  1,  1],
                [ 1,  1,  1],
                [ 3,  1,  1],
                [-1, -1, -1]])])
example_target = np.array([[1, 0, 0, 0, 0, 0,],
                [0, 0, 0, 0, 1, 0,],
                [0, 0, 0, 0, 0, 1,]])

model.fit(example_array, example_target, epochs=1)

Check this answer for a similar question.