2
$\begingroup$

If I'm performing a text classification task using a model built in Keras, and for example, am attempting to predict the appropriate tag given a Stack Overflow question:

"how to subtract 1 from an int how do i subtract 1 from an int?"

And the true/gold tag for this question is:

"objective-c"

But my model is predicting:

"c#"

If I were to retrain my model but this time add the above question and tag in both the training and testing data, would the model be guaranteed to predict the correct tag for this question in the test data?

I suppose the tl;dr is: Are neural networks deterministic if they encounter identical data during training and testing?

I'm aware it's not a good idea to use the same data in both training and testing, but I'm interested from a hypothetical perspective, and for gaining more insight into how neural networks actually learn. My intuition for this question is "no", but I'd really be interested in being pointed to some relevant literature that expands/explains that intuition.

$\endgroup$
2
$\begingroup$

After training, all standard models are deterministic (the process each input goes thru is set).

In essence, during training the model attempts to learn the distribution of the training dataset. Whether it is able to depends on the size of the model, if it is big enough, it can simply "memorize" all the training samples and result in perfect accuracy on the training set.

Normally this is considered to be terrible (called overfitting) and many regularization techniques attempt to prevent it. Eventually when training a model, you are giving it the training distribution as an example but you hope that it will be able to estimate the real distribution out of it.

$\endgroup$
2
$\begingroup$

No, Neural Networks do not have such a guarantee. In fact, I don't believe any kind of classifier in the entire field of Machine Learning has such a guarantee, though some may be slipping my mind...

For an easy counterexample, consider what happens if you have two instances with precisely identical inputs, but different output labels. If your classifier is deterministic (in the sense that there is no stochasticity in the procedure going from input to output after training), which a Neural Network is (unless, for example, you mess up a Dropout implementation and accidentally also apply dropout after training), it cannot possibly generate the correct output for both of those instances, even if they were presented as examples thousands of times during training.

Of course the above is an extreme example, but similar intuition applies to more realistic cases. There can be cases where getting the correct prediction on one instance would reduce the quality of predictions on many other instances if they have somewhat similar inputs. Normally, the training procedure would then prefer getting better predictions on the larger number of instances, and settle for failure on another instance.

$\endgroup$
  • $\begingroup$ I would be tempted to edit this answer to just highlight the 'equal inputs with opposite labels' section, since that is a very clear and elegant verbal proof. $\endgroup$ – DrMcCleod Jan 12 at 14:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.