Output shape of keras.layers.Embedding

Question

I was watching a video on datacamp to learn about Keras, and the instructor used layers.Embedding with keras.layers.Flatten, but he did not really explain the output of the Embedding function properly. I have googled it for like 3-4 hours and could not find anything that could help me.

Objective explained in words: So here is what he was trying to do in normal English. The input data consists of a bunch of college basketball team IDs. Since the ID does not really tell us anything about the team, he used Embedding with an input shape of 1, output shape of 1, and an input dimension equal to the number of teams. He then said that our neural network would "learn" in such a way to associate this new output from embedding as a rating of the basketball team which would be helpful to predict the outcome if two teams had not previously played against each other. Then, (and here is where I got lost) he said that Embedding would actually add an extra dimension to the array so we would have to flatten using keras.layers.flatten().

As I understand it, if I input a team id into Embedding, the output (once the neural network has learned all its parameters) would be ([team_id],[team_rating]), and after flattening, it would be ([team_id,team_rating]). However, the way he described it, after flattening, there would be only one output number: the team_rating. This was especially implied when he added a subtracting layer which would subtract two outputs from this flattened layer (giving the team_rating difference) which would be used to predict the outcome of the game.

Here is the full code from input layer to output layer (Note you probably don't need to read the code to answer the question, but it helps add context)

n_teams = unique(games_season['team_1']).shape[0]

# Create an embedding layer
team_lookup = Embedding(input_dim=n_teams,
                        output_dim=1,
                        input_length=1,
                        name='Team-Strength')

# Create an input layer for the team ID
teamid_in = Input(shape=(1,))

# Lookup the input in the team strength embedding layer
strength_lookup = team_lookup(teamid_in)

# Flatten the output
strength_lookup_flat = Flatten()(strength_lookup)

# Combine the operations into a single, re-usable model
team_strength_model = Model(teamid_in, strength_lookup_flat, name='Team-Strength-Model')

# Input layer for team 1
team_in_1 = Input((1,),name='Team-1-In')

# Separate input layer for team 2
team_in_2 = Input((1,),name='Team-2-In')

# Lookup team 1 in the team strength model
team_1_strength = team_strength_model(team_in_1)

# Lookup team 2 in the team strength model
team_2_strength = team_strength_model(team_in_2)

# Create a subtract layer using the inputs from the previous exercise
score_diff = Subtract()([team_1_strength, team_2_strength])

# Subtraction layer from previous exercise
score_diff = Subtract()([team_1_strength, team_2_strength])

# Create the model
model = Model([team_in_1, team_in_2], score_diff)

---

What I don't get is

A. The input into Embedding/Flatten is a team ID, but isn't the output a list consisting of both the team_id AND the team_rating (since embedding adds an extra dimension(team_rating), and flatten brings the value in that dimension to the same dimension as the original input(team_id). The instructor was passing it off as if there would only be one output: the team_rating

B. If the output actually is a list of both the team_id and the team_rating, wouldn't we want to pick out ONLY the team_rating for future processing such as subtraction of team_rating between different teams?

Any help would be greatly appreciated. I've already tried checking the input shape and output shape of every layer, but for most of them I just get (?,?)

Answer 1

The output of the Embedding layer is a 2D Matrix with one embedding for each ID in the input sequence of teams, by multiplying this matrix with the one hot vector encoding that represents the index of the team this will result in a vector that represents the team_rating

So when the instructor said that it adds a new dimension he didn't mean ([team_id],[team_rating]) . He meant that the embedding layer converts a 1D input which is the ID into 2D representation which is the embedding vector.

So why the flattening? the output of the multiplication between the embedding matrix and the one hot vector(assuming the dimensions in the prev picture) will be V*1 but we need it to be 1*v so he flatten it.