Hot answers tagged

24

The following picture that you used in your question, very accurately describes what is happening. Remember that each element of the 3D filter (grey cube) is made up of a different value (3x3x3=27 values). So, three different 2D filters of size 3x3 can be concatenated to form this one 3D filter of size 3x3x3. The 3x3x3 RGB chunk from the picture is ...


15

In a convolutional neural network, is there a unique filter for each input channel or are the same new filters used across all input channels? The former. In fact there is a separate kernel defined for each input channel / output channel combination. Typically for a CNN architecture, in a single filter as described by your number_of_filters parameter, ...


12

You shouldn't assign all to 0.5 because you'd have the "break symmetry" issue. http://www.deeplearningbook.org/contents/optimization.html Perhaps the only property known with complete certainty is that the initial parameters need to “break symmetry” between different units. If two hidden units with the same activation function are connected to the ...


8

In reverse order to how you asked: all units in a layer become equal since initially the errors due to all of them are the same and thus we train them to be equal This actually happens if you initialise the weights equally (e.g. all zero). Gradients in that case are the same to each neuron in the same layer, and everything changes in lockstep. A neural ...


7

The initial weights in a neural network are initialized randomly because the gradient based methods commonly used to train neural networks do not work well when all of the weights are initialized to the same value. While not all of the methods to train neural networks are gradient based, most of them are, and it has been shown in several cases that ...


7

I'm following up on the answers above with a concrete example in the hope to further clarify how the convolution works with respect to the input and output channels and the weights, respectively: Let the example be as follows (wrt to 1 convolutional layer): the input tensor is 9x9x5, i.e. 5 input channels, so input_channels=5 the filter/kernel size is 4x4 ...


4

Yes, it is not unusual to omit the bias by adding a neuron which always outputs a constant 1, which will then be multiplied by an appropriate weight to give the same formula as you would get using an explicit bias. One notable text using this convention is Understanding Machine Learning: From Theory to Algorithms by Shai Shalev-Shwartz and Shai Ben-David. ...


4

Yes, you can fix (or freeze) some of the weights during the training of a neural network. In fact, this is done in the most common form of transfer learning (which is described here). I don't know exactly how this affects learning in general. In transfer learning, this is definitely beneficial, as we are freezing the weights that are associated with the ...


3

In general, many of the parameters you mentioned are called hyperparameters. All hyperparameters are user-adjusted (or user-programmed) in training phase. Some hyperparameters are: learning rate, batch size, epochs, optimizer, layers, activation functions etc. To answer your (a) part of your question, there are obsiously many frameworks and libraries, for ...


3

tl;dr The equivalent to a neuron in a Fully-Connected (FC) layer is the kernel (or filter) of a Convolution layer Differences The neurons of these two types of layers have two key differences. These are that the convolution layers implement: Sparse connectivity, i.e. each neuron is connected only to an area of the input, not the whole. Weight sharing, i.e. ...


3

There are several ways to answer this question. First of all, there are several mathematical arguments on why using some kind of initialization is better. Consider reading, for example, Xavier et al.. Moreover, there are several numerical experiments showing the importance of initialization. The motivation for Xavier initialization in Neural Networks is to ...


3

Hi and welcome to the community. It's important to understand these basic concepts very clearly. You have to first understand the basic unit of a neural network, a single node/neuron/perceptron. Let us forget all about Neural Networks for a bit, and talk about something far simpler. Linear Regression In the above figure, we clearly have one independent ...


3

tl;dr The whole point of gradient descent is to assess the contribution of each parameter towards the loss. This information is uncovered through the gradient of the loss w.r.t each parameter. A deeper look... Suppose we have a NN with parameters $w_{i}, \; i={1, 2, ...}$. This NN makes some predictions, which we compare to the actual targets and compute a ...


2

If the task involves only apples, orange and peaches, you should use method 1. As the number of classes is small, the network cannot generalize well to all classes. As a side note, you should start with the pretrained weights of YOLO v3 as some classes of YOLO v3 may be fruits, which can help your model converge faster. If the number of classes is large, ...


2

Nope! Our number of coefficients will be driven by the vocabulary, and we'll use each of those 10K samples to estimate values for those coefficients - so, 'just' 100K samples. However, word frequency in human languages follows a Zipf distribution => most of those words will be rare, seen in only a few samples (=> won't even be able to determine whether ...


2

That is a very deep question. There was series of papers recently proving the convergence of gradient descent for overparameterized deep networks (for example, Gradient Descent Finds Global Minima of Deep Neural Networks, A Convergence Theory for Deep Learning via Over-Parameterization or Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU ...


2

Sorry, this is a very broad area. Proper understanding of neural networks requires advanced mathematics. It's not sufficient to say "balancing of weights and biases" because most ML algorithms have weights. You seriously need to grab a book. OCR system itself is also very broad, it includes various object recognition techniques. You haven't even mentioned ...


2

Regularizer's are used as a means to combat over fitting.They essentially create a cost function penalty which tries to prevent quantities from becoming to large. I have primarily used kernel regularizers. First I try to control over fitting using dropout layers. If that does not do the job or leads to poor training accuracy I try the Kernel regularizer. I ...


2

If I understood correctly, the model is a polynomial equation No, it's not true that all machine learning (ML) models compute (or represent) a polynomial function. For example, a sigmoid is not a polynomial, but, for example, in a neural network, you can combine many sigmoids to build complicated functions that may not necessarily be polynomials. We usually ...


2

The conventions I have seen tend to post-multiply rather than pre-multiply, although there are examples in the literature which adopt the opposite convention. Some examples include: In Deep Learning: An Introduction for Applied Mathematicians, a layer with input $x \in \mathbb R^n$ and output $f(x) \in \mathbb R^m$ is computed by $$ f(x) = \sigma(Wx + b)$$ ...


2

Is it trying to make sure there is no symmetry in the gradients? The aim of weight initialization is to make sure that we don't converge to a trivial solution. That's why we have different kinds of initialization depending on the dataset type. So, Yes it is trying to avoid symmetry. Is it trying to allow the gradients to be large so it can quickly converge?...


2

is it common to deal with weights and biases in everyday tasks or in most of the cases existing algorithms do it well? No; and it is no coincidence that you will not be able to find any reference to such a practice in any course or tutorial about neural networks. Such a practice would require a whole additional level of (business/SME) know-how in order to ...


2

Interesting question, I can come with 2 explanations why we don't initialize weights with 1 mean value : It may be easier for the network to learn identity function, but we may have a similar issue about not being able to learn comparison, comparison is quite an important reasoning in my opinion, this is why having negative weight values is important, and ...


1

No, it does not take into account the curvature. But, if curvature is important for you, then, it would be a good idea to look at Ricci flow and its applications in neural networks.


1

The most important thing we achieve is indeed making sure the weights are not all equal. If they were, every layer would behave as if it were a single cell. We typically want weights that are near zero (so unimportant connections will not accidentally dominate) but non-zero. The different types of initialization all have different motivations, including ...


1

There are many resources that answer your question, but, given that you're apparently new to machine learning (ML), deep learning (DL), and neural networks (NN), let me provide a simple answer that should clarify your doubts. The term weight in the context of ML, DL, and NN is a synonym for parameter (sometimes, in some contexts, such as linear regression, ...


1

The paper A systematic study of the class imbalance problem in convolutional neural networks is a great overview on class imbalance approaches. Section 2 summarizes various methods commonly used. They categorize "Adding Class Weights for an imbalanced dataset" under the technique "Cost sensitive learning": Cost sensitive learning. This ...


1

Weights are not normally shared across Transformer layers in vanilla Transformers. However, there has been research done in testing out sharing weights, and sometimes they improve the scores. Here are some examples: ALBERT is an improvement on BERT (so only uses the encoding side, no decoder), and shows that sharing the attention weights only $\left\{ W_i^Q, ...


1

I'll use notation from the paper you cited, and any other readers should refer to the paper (widely available) for definitions of notation. The utility of using $W^Q$ and $W^K$, rather than $W$, lies in the fact that they allow us to add fewer parameters to our architecture. $W$ has dimension $d_{model} \times d_{model}$, which means that we are adding $d_{...


1

Actually, I just started inspecting the entries further down in the leaderboard list, and there are in fact more modest architectures, e.g. this one, which uses 3 hidden layers with 8 units each.


Only top voted, non community-wiki answers of a minimum length are eligible