35
votes
Accepted
Why do CNN's sometimes make highly confident mistakes, and how can one combat this problem?
The concept you are looking for is called epistemic uncertainty, also known as model uncertainty. You want the model to produce meaningful calibrated probabilities that quantify the real confidence of ...
- 1,355
15
votes
Why do CNN's sometimes make highly confident mistakes, and how can one combat this problem?
Your classifier is specifically learning the ways in which 0s are different from other digits,
not what it really means for a digit to be a zero.
Philosophically, you could say the model appears to ...
11
votes
Accepted
Effect of batch size and number of GPUs on model accuracy
This should make a difference, but how big is the difference heavily depends on your task. However, generally speaking, a smaller batch size will have a lower speed if counted in sample/minutes, but ...
- 1,745
9
votes
Can LSTM neural networks be sped up by a GPU?
From Nvidia www (https://developer.nvidia.com/discover/lstm):
Accelerating Long Short-Term Memory using GPUs
The parallel processing capabilities of GPUs can accelerate the LSTM training and ...
- 1,293
7
votes
Why do CNN's sometimes make highly confident mistakes, and how can one combat this problem?
Broken assumptions
Generalization relies on making strong assumptions (no free lunch, etc). If you break your assumptions, then you're not going to have a good time. A key assumption of a standard ...
- 893
6
votes
Accepted
Can LSTM neural networks be sped up by a GPU?
I found that there are cuDNN accelerated cells in Keras, for example, https://keras.io/layers/recurrent/#cudnnlstm.
They are very fast. The normal LSTM cells are faster on CPU than on GPU.
- 289
6
votes
Accepted
Deep Q-Learning poor convergence on Stochastic Environment
The inputs that you describe seem like they should be sufficient for a DQN-based agent to learn a good strategy for playing Minesweeper, regardless of whether or not the starting layout changes. The ...
- 10.4k
4
votes
Accepted
Accuracy too high too fast?
It actually depends on a couple of things here -
How many output classes do you have? If you have only 2 or 3 classes, it is a very easy task for the classifier that you have built. So, it is highly ...
- 158
4
votes
Why would you implement the position-wise feed-forward network of the transformer with convolution layers?
I'm going to post another guess to this question - it won't be a complete answer, but hopefully it'll provide some direction towards finding a more legitimate answer.
The feed-forward networks as ...
- 91
4
votes
Accepted
Why are traditional ML models still used over deep neural networks?
Why are still traditional machine learning (ML) models used over neural networks if neural networks seem to be superior to traditional ML models?
Of course, the model that achieves state-of-the-art ...
- 41.1k
3
votes
Accepted
How can I prevent the CNN from classifying a new input into one of the existing labels (it was trained with) when the input has a new different label?
You can introduce another class to your network - "not a book". After that, you will need to add new data to your dataset, random images that do not contain books to classify and train your network on ...
- 276
3
votes
How to constraint the output value of a neural network?
There are many ways of constraining the network's output.
Using an activation layer is a good one. If you sigmoid the output layer, the output is constrained between [0,1] and you can multiply that by ...
- 906
3
votes
Will a neural network always predict the correct label if it sees the exact same input during training and testing?
No, Neural Networks do not have such a guarantee. In fact, I don't believe any kind of classifier in the entire field of Machine Learning has such a guarantee, though some may be slipping my mind...
...
- 10.4k
3
votes
Adding BERT embeddings in LSTM embedding layer
Instead of using the Embedding() layer directly, you can create a new bertEmbedding() layer and use it instead.
...
- 403
3
votes
Entropy term in Proximal Policy Optimization (PPO) becomes undefined after few training epochs
I browsed through some other implementations of PPO and they all add small offset (1e-10) to prevent undefined log(0). I did that and the training works now.
- 191
3
votes
Why would you implement the position-wise feed-forward network of the transformer with convolution layers?
1) The math is the exact same, so from an optimization or mathematical perspective there is no difference
2) Here are my guesses to a possible answer.
Habit: People may just call one over the ...
- 2,379
3
votes
Accepted
Is CNN capable of extracting the descriptive statistics features
CNNs learn convolutional filters that get trained on finding local, recurring patterns in some kind of image/volume data. 1D convolution is actually a thing, but I think what would be more suitable ...
- 825
3
votes
Accepted
How to describe an keras Model in a scientific report
Some other details you could mention are:
total number of model parameters (e.g. 1.2M or 0.15M) & depth of the network (e.g. 38-layered network)
family/style of the network architecture (e.g. ...
- 339
3
votes
Effect of batch size and number of GPUs on model accuracy
No. Different batch sizes mean different gradients (check stochastic gradient descent concept you will get how loss calculated) are calculated in each step, and thus the gradient descent will likely ...
- 224
3
votes
Why are traditional ML models still used over deep neural networks?
This question is very broad, so let me attempt to answer it using my own background in time series analysis.
As an example, why would I continue using ARIMA to forecast a time series? Why not simply ...
- 151
3
votes
Why do CNN's sometimes make highly confident mistakes, and how can one combat this problem?
Apollys,
That's a very well thought out response. Particularly, the philosophical discussion of the essence of "0-ness."
I haven't actually performed this experiment, so caveat emptor... I wonder ...
- 131
3
votes
Accepted
Is my GRU model under-fitting given this plot of the training and validation loss?
When ever you are buliding a ML Model don't take accuracy seriously(Mistake done by Netflix that cost them alot), you should try to get the hit scores as they will help you to know how many times your ...
- 70
3
votes
Accepted
Why MLP cannot approximate a closed shape function?
In neural networks, the family of functions and the shapes that they can make for decision surfaces is determined by the activation function you use (in your case, ...
- 32.9k
3
votes
Accepted
What is the need for so many filters in a CNN?
Then how do each filter differ by? Is it in hovering over the input matrix? Or is it in the values contained by filter itself? Or differs in both hovering and content?
The filters (aka kernels) are ...
- 41.1k
3
votes
Accepted
Can predictions of a neural network using ReLU activation be non-linear (i.e. follow the pattern) outside of the scope of trained data?
It isn't too surprising to see behaviour like this, since you're using $\mathrm{ReLU}$ activation.
Here is a simple result which explains the phenomenon for a single-layer neural network. I don't have ...
- 1,010
3
votes
What is the proper way to process continuous sequence data, such as time-series, using the Transformer?
Instead of using a token embedding you can use a linear layer. For an input of (10, 5, 4) - (sequence length, batch size, features) you can create a linear layer:
...
- 283
3
votes
Accepted
Are these book example CNN results realistic?
I was able to run the code without "any" modifications on Tensorflow 2.4.0, just had to replace the imports:
...
- 787
3
votes
Accepted
Weights initialization once the Neural Network is trained
Regarding your first code snippet, there is no weight storing or continuation of training between the different CV folds whatsoever: each model is trained anew with the respective training data of ...
- 1,040
3
votes
How embeddings learned from one model can be used in another?
To answer your one question: Are embeddings model-specific? YES! They are. I am not going to invoke math or other techniques here. My explanation is going to be in a intuitive perspective. I don't ...
- 131
3
votes
Why is a simple regression problem so hard for an MLP to learn?
An interesting problem. This network has only 933 trainable parameters, and obtains MeanAbsolutePercentageError of 0.01 - 0.04. It is based on a softmax activation, ...
- 787
Only top scored, non community-wiki answers of a minimum length are eligible