0
$\begingroup$

I want to use a pretrained model found in [BERT Embeddings] https://github.com/UKPLab/sentence-transformers and I want to add a layer to get the sentence embeddings from the model and pass on to the next layer. How do I approach this?

The inputs would be an array of documents and each document containing an array of sentences.

The input to the model itself is a list of sentences where it will return a list of embeddings.

This is what I've tried but couldn't solve the errors:

def get_embeddings(input_data):

    input_embed = []
    for doc in input_data:
      doc = tf.unstack(doc)
      doc_arr = asarray(doc)
      doc = [el.decode('UTF-8') for el in doc_arr]
      doc = list(doc)
      assert(type(doc)== list)

      new_doc = []
      for sent in doc:
        sent = tf.unstack(sent)
        new_doc.append(str(sent))
        assert(type(sent)== str)

      embedding= model.encode(new_doc)  # Accepts lists of strings to return BERT sentence embeddings
      input_embed.append(np.array(embedding))

    return tf.convert_to_tensor(input_embed, dtype=float)


sentences = tf.keras.layers.Input(shape=(3,5)) #test shape
sent_embed = tf.keras.layers.Lambda(get_embeddings)


x = sent_embed(sentences)

Errors

$\endgroup$

1 Answer 1

1
$\begingroup$

I think you should use Keras embedding layer. It will be too easier than what you are doing.

Steps

  • Create Embedding Matrix
  • add matrix to embedding layer while building model.

You will find detailed article

https://www.cs.uaf.edu/2011/spring/cs641/lecture/04_05_modeling.html

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .