I am planning to use BERT embeddings in the LSTM embedding layer instead of the usual Word2vec/Glove Embeddings. What are the possible ways to do that?

  • $\begingroup$ do you want the entire bert contextual embedding or just the subword embeddings? $\endgroup$
    – mshlis
    Jun 17 '19 at 15:27
  • $\begingroup$ I would need the contextual embeddings. $\endgroup$ Jun 18 '19 at 5:07

Instead of using the Embedding() layer directly, you can create a new bertEmbedding() layer and use it instead.

# Sample code
# Model architecture

# Custom BERT layer
bert_output = BertLayer(n_fine_tune_layers=10)(bert_inputs)

# Build the rest of the classifier 
dense = tf.keras.layers.Dense(256, activation='relu')(bert_output)
pred = tf.keras.layers.Dense(1, activation='sigmoid')(dense)

model = tf.keras.models.Model(inputs=bert_inputs, outputs=pred)
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

This article will walk you through the entire process of creating the custom BERT layer along with example code. Give it a read.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.