0
$\begingroup$

I want to use a pretrained model found in [BERT Embeddings] https://github.com/UKPLab/sentence-transformers and I want to add a layer to get the sentence embeddings from the model and pass on to the next layer. How do I approach this?

The inputs would be an array of documents and each document containing an array of sentences.

The input to the model itself is a list of sentences where it will return a list of embeddings.

This is what I've tried but couldn't solve the errors:

def get_embeddings(input_data):

    input_embed = []
    for doc in input_data:
      doc = tf.unstack(doc)
      doc_arr = asarray(doc)
      doc = [el.decode('UTF-8') for el in doc_arr]
      doc = list(doc)
      assert(type(doc)== list)

      new_doc = []
      for sent in doc:
        sent = tf.unstack(sent)
        new_doc.append(str(sent))
        assert(type(sent)== str)

      embedding= model.encode(new_doc)  # Accepts lists of strings to return BERT sentence embeddings
      input_embed.append(np.array(embedding))

    return tf.convert_to_tensor(input_embed, dtype=float)


sentences = tf.keras.layers.Input(shape=(3,5)) #test shape
sent_embed = tf.keras.layers.Lambda(get_embeddings)


x = sent_embed(sentences)

Errors

$\endgroup$
1
$\begingroup$

I think you should use Keras embedding layer. It will be too easier than what you are doing.

Steps

  • Create Embedding Matrix
  • add matrix to embedding layer while building model.

You will find detailed article

https://www.cs.uaf.edu/2011/spring/cs641/lecture/04_05_modeling.html

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.