2
$\begingroup$

I am planning to use BERT embeddings in the LSTM embedding layer instead of the usual Word2vec/Glove Embeddings. What are the possible ways to do that?

Improve this question
$\endgroup$
2
  • $\begingroup$ do you want the entire bert contextual embedding or just the subword embeddings? $\endgroup$
    – mshlis
    Jun 17, 2019 at 15:27
  • $\begingroup$ I would need the contextual embeddings. $\endgroup$
    – Srikant Jayaraman
    Jun 18, 2019 at 5:07

1 Answer 1

Reset to default
3
$\begingroup$

Instead of using the Embedding() layer directly, you can create a new bertEmbedding() layer and use it instead.

# Sample code
# Model architecture

# Custom BERT layer
bert_output = BertLayer(n_fine_tune_layers=10)(bert_inputs)

# Build the rest of the classifier 
dense = tf.keras.layers.Dense(256, activation='relu')(bert_output)
pred = tf.keras.layers.Dense(1, activation='sigmoid')(dense)

model = tf.keras.models.Model(inputs=bert_inputs, outputs=pred)
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(...)

This article will walk you through the entire process of creating the custom BERT layer along with example code. Give it a read.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .