0
$\begingroup$

The output dimensions of BERT are 768-dimensional, is it possible to reduce them to a lower, custom number? For example, if there is another BERT-based transformer model which has a lower count of ouput dimensions, if it's possible to fine tune BERT on MLM to output lower dimension encodings etc.

And if not, are there any possible workarounds for this issue?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

SBERT.net gives an example that does exactly (or very close to) what you requested. The approach is based on PCA that projects high dimensional vectors (768 dim in this case) to lower dimensions with some performance drop. It even mentioned that it's possible to reduce the original vector to dimension as low as 32.

For more details, please refer to Model Distillation

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .