I have a huge networt ( keras-bert ) which works fine for classification. Since my data has two different columns, I'd like to fine-tune a BERT model for each column and connect them in the final layer. But I get the following error:
---> 20 model = keras.models.Model(inputs=[inputs1, inputs2], outputs=outputs)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/network.py in _validate_graph_inputs_and_outputs(self)
1620 """Validates the inputs and outputs of a Graph Network."""
1621 # Check for redundancy in inputs.
-> 1622 if len(set(self.inputs)).= len(self:inputs):
1623 raise ValueError('The list of inputs passed to the model '
1624 'is redundant. '
TypeError: unhashable type: 'list'
In my code I have two bert models, model1
and model2
. With just one model it worked fine. The only things I added were that 2 models instead of one are loaded from checkpoint and the second input-layer and the concatenation of dense1 and dense2:
#load_trained_model_from_checkpoint is defined here:
# https://github.com/CyberZHG/keras-bert/blob/master/keras_bert/loader.py
model1 = load_trained_model_from_checkpoint(
config_path,
checkpoint_path,
training=True,
trainable=True,
seq_len=SEQ_LEN,
)
model2 = load_trained_model_from_checkpoint(
config_path,
checkpoint_path,
training=True,
trainable=True,
seq_len=SEQ_LEN,
)
inputs1 = model1.inputs[:2] #model 1 for titles
inputs2 = model2.inputs[:2] #model 2 for texts
dense1 = model1.get_layer('NSP-Dense').output
dense2 = model2.get_layer('NSP-Dense').output
outputs = keras.layers.Dense(len(test_title_y[0]), activation='sigmoid')(keras.layers.concatenate([dense1, dense2]))
model = keras.models.Model(inputs=[inputs1, inputs2], outputs=outputs)
What am I overseeing? Do I somehow have to wrap the input?
Edit: I suspect that the problem has something to do with my input being a list of lists: the inputs1 and inputs2 look like that:
[<tf.Tensor 'Input-Token:0' shape=(?, 256) dtype=float32>, <tf.Tensor 'Input-Segment:0' shape=(?, 256) dtype=float32>]
[<tf.Tensor 'Input-Token_1:0' shape=(?, 256) dtype=float32>, <tf.Tensor 'Input-Segment_1:0' shape=(?, 256) dtype=float32>]
Can I somehow reshape or concatenate my input to overcome this error?
Edit:
The summaries of model1 looks like that, model2 looks the same but with LAYER_2 for each layer name:
Layer (type) Output Shape Param # Connected to
Input-Token (InputLayer) (None, 256) 0
Input-Segment (InputLayer) (None, 256) 0
Embedding-Token (TokenEmbedding [(None, 256, 768), ( 23440896 Input-Token[0][0]
Embedding-Segment (Embedding) (None, 256, 768) 1536 Input-Segment[0][0]
... (lots of layers in between)
NSP-Dense (Dense) (None, 768) 590592 Extract[0][0]
It looks like there is an internal check which looks for equal length inputs to a model. In this case both inputs are [batchSize, 256]
. One thing that you can do is try packing the two inputs into one with three dimensions (ie [batchSize, inputs, 256]
). You would need to handle some slicing in the model to make sure you are passing the correct input to the layers.
Here is an example:
#load_trained_model_from_checkpoint is defined here:
# https://github.com/CyberZHG/keras-bert/blob/master/keras_bert/loader.py
model1 = load_trained_model_from_checkpoint(
config_path,
checkpoint_path,
training=True,
trainable=True,
seq_len=SEQ_LEN,
)
model2 = load_trained_model_from_checkpoint(
config_path,
checkpoint_path,
training=True,
trainable=True,
seq_len=SEQ_LEN,
)
inputs1 = model1.inputs[:2] #model 1 for titles
inputs2 = model2.inputs[:2] #model 2 for texts
# concatenate the inputs into one tensor
concatInputs = keras.backend.concatenate([inputs1, inputs2],axis=0)
dense1 = model1.get_layer('NSP-Dense').output
dense2 = model2.get_layer('NSP-Dense').output
def getModel():
inputs=keras.layers.Input([2,256])
# slice inputs into individual tensors
in1 = keras.layers.Lambda(lambda x: tf.slice(inputs,[0,0],[1,inputs.get_shape()[1]])
in2 = keras.layers.Lambda(lambda x: tf.slice(inputs,[1,0],[2,inputs.get_shape()[1]])
d1 = dense1(in1)
d2 = dense2(in2)
outputs = keras.layers.Dense(len(test_title_y[0]), activation='sigmoid')(keras.layers.concatenate([d1, d2]))
return inputs,outputs
ins,outs = getModel()
model = keras.models.Model(inputs=[ins], outputs=[outs])
I didn't get a chance to test so the syntax may not be exact but the idea should work.
This may sound a little copy+paste, but I think you solved your own question when you noticed the inputs are lists of lists.
Instead of using [inputs1, inputs2]
, use inputs1 + inputs2
.
Test if they are lists with isinstance(inputs1, list)
.
The error means that somewhere, the function is trying to use a list as a hash argument instead. ie using a list as a key in a dictionary.
The Model(inputs=[input1,input2]) cannot take in a list.
Try using
np.reshape
On input1 and input2 first. Additionally, cast the lists to tuples.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.