Keras中的尺寸错误

Question

I am tying to implement a simple word2vec model but I get the following error

ValueError: Error when checking target: expected dense-softmax to have 3 dimensions, but got array with shape (32, 14).

the variable train_x and train_y are 32 lines of the form

[[0 0 0 0 0 0 0 0 0 1 0 0 0 0]
 [0 0 0 0 1 0 0 0 0 0 0 0 0 0]
 [0 0 0 0 1 0 0 0 0 0 0 0 0 0]
                          ...]]

and the python code is the following

vocal_size = 14
input = Input(shape=(vocal_size, ), dtype='int32', name='input')
embeddings = Embedding(output_dim=5, input_dim= vocal_size)(input)
output = Dense(vocal_size, use_bias=False, activation='softmax')(embeddings)
model = Model(input=input, output=output)
model.compile(optimizer='adam', loss='categorical_crossentropy')
model.summary()
model.fit(train_x, train_y)



_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input (InputLayer)           (None, 14)                0         
_________________________________________________________________
embeddings (Embedding)       (None, 14, 5)             70        
_________________________________________________________________
dense_1 (Dense)              (None, 14, 14)            70        
=================================================================
Total params: 140
Trainable params: 140
Non-trainable params: 0

Edit:

("I like stackoverflow") with context size 1, I create the following tuples,
("I", "like"), ("like", "I") , ("like", "stackoverflow"), ("stackoverflow", "like")

Then I do an one-hot encoding of all of them and feed them to the model.

train_x[0] -> is one hot encoding of the word "I"
train_y[0] -> is one hot encoding of the context word "like"

Edit 2

Using the first encoding for skip-gram: Treating 0 as a special word (ie not top 10.000 most frequent) and start the counting from 1. I assume I should give as an input a single number and output a one-hot encoding ie ("stack", "overflow"), input [3] ("stack") and the output [0,0,0,0,1,0,0,0,0,0,0] ("overflow").

Input(shape=(1,)..) -> 
Embedding(output_dim=embedding_size, input_dim=vocab_size, mask_zero=True, ...) -> 
Dense(vocab_size+1, activation="Softmax")
model.compile(optimizer='SGD', loss='categorical_crossentropy')

Ie embedding_size = 5, input the sentences in your example,

https://imgur.com/a/32m4z

Answer 1

Thanks for the edit. You're running into trouble for two reasons, one shallow, one deep. First: shallow, the dense layer requires a three-dimensional input, but the embedding is two dimensional. You can fix this with a Flatten :

input = Input(shape=(vocal_size, ), dtype='int32', name='input')
embeddings = Embedding(output_dim=5, input_dim=vocal_size+1, input_length=vocal_size)(input)
flat = Flatten(embeddings)
output = Dense(vocal_size, use_bias=False, activation='softmax')(flat)

The deep is because one-hot encoding and embedding are two options that serve the same purpose, so you don't need both (see here and here ).

The embedding layer wants a series of "sentences" made of integers representing words (or tuples) and a vocabulary size, so something like

['Welcome to stack overflow',
'stack overflow is great',
'Hope it's helpful to you']

would be represented as

[[1,2,3,4,0],[3,4,5,6,0],[7,8,9,2,10]] 
# 0s are there to "pad" sentences 1 & 2 as they all need to be the same length

and fed into an embedding layer like this:

input = Input(shape=(5, ), dtype='int32')
embeddings = Embedding(output_dim=5, input_dim=11, input_length=5)(input)
#input dim is 11 because we want 1 more than the number of words in our vocabulary
#padding can be done with the keras function pad_sequences

I'm certain you know, the one hot encoding of our sentences would look like this:

[[1,1,1,1,0,0,0,0,0,0],
 [0,0,1,1,1,1,0,0,0,0],
 [0,1,0,0,0,0,1,1,1,1]]

Because the sentences have already been transformed (one hot has "embedded" our sentences as binary vectors in 10-dimensional space), we can feed this directly into a Dense layer without the need for further embedding:

input = Input(shape=(vocal_size, ), dtype='int32', name='input')
output = Dense(vocal_size, use_bias=False, activation='softmax')(input)

Here's a functional toy example using both ways:

from keras.layers import Dense,Activation,Embedding,Input,Flatten
from keras import Model
import numpy as np

wrords = ['Welcome to stack overflow',
    'stack overflow is great',
    'Hope it\'s helpful to you']

a = [[1,2,3,4,0],[3,4,5,6,0],[7,8,9,2,10]]
b = [[1,1,1,1,0,0,0,0,0,0],
 [0,0,1,1,1,1,0,0,0,0],
 [0,1,0,0,0,0,1,1,1,1]]
c = [1,1,0] #hypothetical target is "references stack overflow"

input = Input(shape=(5, ), dtype='int32', name='input')
embeddings = Embedding(output_dim=5, input_dim=11, input_length=5)(input)
flat = Flatten()(embeddings)
output = Dense(1, activation='softmax')(flat)
model = Model(input=input, output=output)
model.compile(optimizer='adam', loss='binary_crossentropy')
model.summary()
model.fit(np.array(a),np.array(c))

input2 = Input(shape=(10, ), dtype='float32')
output2 = Dense(1, activation='softmax')(input2)
model2 = Model(input=input2, output=output2)
model2.compile(optimizer='adam', loss='binary_crossentropy')
model2.summary()
model2.fit(np.array(b),np.array(c))