3
$\begingroup$

If I'm performing a text classification task using a model built in Keras, and, for example, I am attempting to predict the appropriate tag for a given Stack Overflow question:

How do I subtract 1 from an integer?

And the ground-truth tag for this question is:

But my model is predicting:

If I were to retrain my model, but this time add the above question and tag in both the training and testing data, would the model be guaranteed to predict the correct tag for this question in the test data?

I suppose the tl;dr is: Are neural networks deterministic if they encounter identical data during training and testing?

I'm aware it's not a good idea to use the same data in both training and testing, but I'm interested from a hypothetical perspective, and for gaining more insight into how neural networks actually learn. My intuition for this question is "no", but I'd really be interested in being pointed to some relevant literature that expands/explains that intuition.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
3
$\begingroup$

No, Neural Networks do not have such a guarantee. In fact, I don't believe any kind of classifier in the entire field of Machine Learning has such a guarantee, though some may be slipping my mind...

For an easy counterexample, consider what happens if you have two instances with precisely identical inputs, but different output labels. If your classifier is deterministic (in the sense that there is no stochasticity in the procedure going from input to output after training), which a Neural Network is (unless, for example, you mess up a Dropout implementation and accidentally also apply dropout after training), it cannot possibly generate the correct output for both of those instances, even if they were presented as examples thousands of times during training.

Of course the above is an extreme example, but similar intuition applies to more realistic cases. There can be cases where getting the correct prediction on one instance would reduce the quality of predictions on many other instances if they have somewhat similar inputs. Normally, the training procedure would then prefer getting better predictions on the larger number of instances, and settle for failure on another instance.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ I would be tempted to edit this answer to just highlight the 'equal inputs with opposite labels' section, since that is a very clear and elegant verbal proof. $\endgroup$
    – DrMcCleod
    Jan 12, 2019 at 14:02
  • $\begingroup$ In principle, a rule-based or exact classifier could always be right. For instance, if you store the images that you've seen during training or testing/validation in a database, then, when a new image arrives, compare it with all other images in the database, you can simply return the correct label (if such an image exists in the dataset). So, I suppose that you're excluding this type of "classifier" when you say "ML classifier". In any case, the key idea is: most ML models typically approximate the desired function, so there's a chance that they may be wrong. $\endgroup$
    – nbro
    Jan 17, 2021 at 17:25
1
$\begingroup$

After training, all standard models are deterministic (the process each input goes thru is set).

In essence, during training the model attempts to learn the distribution of the training dataset. Whether it is able to depends on the size of the model, if it is big enough, it can simply "memorize" all the training samples and result in perfect accuracy on the training set.

Normally this is considered to be terrible (called overfitting) and many regularization techniques attempt to prevent it. Eventually when training a model, you are giving it the training distribution as an example but you hope that it will be able to estimate the real distribution out of it.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .