2
The most distinct words of a language are usually the function words (the, and, of, with,...); other lexical items are often (at least partly) shared between languages that had come in contact with each other. So looking for function words is usually the best way to identify the language in a given text.
This can be done by having a list of function words ...
1
Answer to Q1) If sampling for next token do you need to apply mask during inference?
Yes you do! The models ability to transfer information across positions was trained in this manner, and changing it up will have unpredictable consequences. Let my try to give an example:
Tokens: 1:sally, 2:sold, 3:seashells, 4:on, 5:the, 6:____
In the above you are ...
1
Division by zero is not mathematically defined.
A usual or standard way of dealing with this issue is to raise an exception. For example, in Python, the exception ZeroDivisionError is raised at runtime if you happen to divide by zero.
If you execute the following program
zero = 0
numerator = 10
numerator / zero
You will get
Traceback (most recent call ...
1
You can use an autoencoder for text. For example, you can refer to this example here: https://machinelearningmastery.com/lstm-autoencoders/
For comparing the contextual similarity, you can compare the encoded vectors for the distance maybe through a formula like mean squared error.
This works as the autoencoder compresses the input data into a vector of ...
Only top voted, non community-wiki answers of a minimum length are eligible
