I am planning to use BERT embeddings in the LSTM embedding layer instead of the usual Word2vec/Glove Embeddings. What are the possible ways to do that?

  • $\begingroup$ do you want the entire bert contextual embedding or just the subword embeddings? $\endgroup$
    – mshlis
    Jun 17, 2019 at 15:27
  • $\begingroup$ I would need the contextual embeddings. $\endgroup$ Jun 18, 2019 at 5:07

1 Answer 1


Instead of using the Embedding() layer directly, you can create a new bertEmbedding() layer and use it instead.

# Sample code
# Model architecture

# Custom BERT layer
bert_output = BertLayer(n_fine_tune_layers=10)(bert_inputs)

# Build the rest of the classifier 
dense = tf.keras.layers.Dense(256, activation='relu')(bert_output)
pred = tf.keras.layers.Dense(1, activation='sigmoid')(dense)

model = tf.keras.models.Model(inputs=bert_inputs, outputs=pred)
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

This article will walk you through the entire process of creating the custom BERT layer along with example code. Give it a read.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .