Tag Info

1

There is at least Swish, which is defined as $f(x) = x \cdot \text{sigmoid}(\beta x)$. ...This suggests that Swish can be loosely viewed as a smooth function which nonlinearly interpolates between the linear function and the ReLU function. The degree of interpolation can be controlled by the model if β is set as a trainable parameter. There is an other ...

1

ReLU, which is widely used in deep learning, is not injective because all negative numbers are mapped to zero. So, either you misunderstood the definition of an injective function or you forgot about ReLU.

0

How/why do we achieve the same/similar results though we are skipping a layer altogether Dropout is not a layer, even tough deep learning libraries implement it as a layer module for convenience. Why do we achieve same results? We don't, that's why dropout is applied only during training and not during test. And the fact that results change is also the core ...

2

The loss graph indicates that the model converged to a local minimum, already after a few epochs, and the weights start to oscillate around it. The learning rate is surely responsible for it, but it's not the only culprit. Reducing the learning rate with a scheduler didn't work in your case most likely because SGD applies the same learning rate to all ...

0

In machine translation, there is a widely used BLEU score ( https://en.wikipedia.org/wiki/BLEU ). It simply counts the matching n-grams between two segments of text and returns a 0-1 score based on that. The problem with this method is that it would give the same score to pairs "It is hot"/"It is cold" and "It is hot" / "It ...

2

Regarding your first code snippet, there is no weight storing or continuation of training between the different CV folds whatsoever: each model is trained anew with the respective training data of each fold and validated on the validation data. Notice that this is exactly the idea behind cross validation - models trained on different folds are completely ...

0

There might be several reasons for that: the data is easily understood by the model you are using the model you use is fitted to the problem the problem complexity is low There are a lot more reasons to explain the convergence of an algorithm.

2

First, I have not read and do not have that book. That said, I would interpret that statement in the context of the intractability of guaranteeing that the optimization function will find global minima in the loss surface. In other words, higher precision values will do nothing to improve whether we have descended into a global or local minimum. On the ...

1

It could be because there is simply not enough data for the late game. To make the model give more importance to the later stages of the game you can try to tweak the loss function such that it penalizes more for when there are fewer pieces on the board. ( This might give an idea on that: https://medium.com/visionwizard/understanding-focal-loss-a-quick-read-...

0

You don't put batch normalization or dropout layers after the last layer, it will just "corrupt" your predictions. They are intended to be used only within the network, to help it converge and avoid overfitting. BTW even if your fully connected layer's output is always positive, it would have positive and negative outputs after batch normalization. ...

1

In Bayesian statistics, as opposed to frequentist statistics, you can model the parameters as random variables. Bayesian machine learning is the application of Bayesian statistics in the context of machine learning. The specific application of Bayesian statistics to learning in neural networks is denoted as Bayesian deep learning. Even just in the simple ...

0

Look, this is currently a quite contentious issue. J3soons answer already links to the original paper and another one. However, most of the evidence for batch normalization is still empirical, and there is very little theoretical explanation for why it works. There are a number of competing theories out there. Also, there are a number of papers providing ...

1

Predicting the correct amount of repetitions for an action sounds like a regression task. Turning it into a classification task using a model with n output nodes will lead to several drawbacks, the biggest ones being: Having to choose a priori a finite max amount of actions n Turning the data into really sparse vectors, especially for large n. So a better ...

2

Let me try to explain here. Usually, we calculate the variance by subtracting the mean term and then square it. But here mean (first-moment m_t) is fluctuating like anything at each time "t" and is getting calculated with the influence of past mean as well, also with the influence of beta_1. So when the 2nd-moment term v_t is getting calculated ...

0

To make the dynamic amount of features easier to work with, you can model it as a sequence modeling problem, where for the new task your sequence length increases. (where each "timestep" is a single feature) Continual learning literature might still be the right place to look, as they try to solve the same problem of catastrophic forgetting that ...

1

This is the case as the loss doesn't have to monotonically decrease when it's updated in the negative direction. For example: Let $L(\theta) = \theta^2$ and $\theta_0= 3$ Let the subscript n in $\theta_n$ denote the iteration number. Then $\nabla_{\theta}L(\theta_0) = 2*\theta = 2*3 = 6$ For the loss to decrease in this case $\epsilon < 1$ needs to hold ...

0

Lateral connection in NN simply mean that units in a hidden layer are connected with one another. Suppose we have a hidden layer Hi which has 10 from (1,2,....10) than lateral connection implies that unit 1 may be connected to unit 2. So now the activation of unit 2 is not only dependent on incoming inputs but also unit 1. The weight assigned to connection ...

0

Surely you can implement such algorithm, since you already know the details. Iterative steps: Determine possible numbers in all empty squares Find the square with the least number of possible numbers Does this square have an unique solution? If yes, set it and GOTO 1 Else apply the backtracking logic and guess a number, GOTO 1 People who are good with ...

0

the issue is that there are too many variables (atomic forces) to consider when simulating how an amino acid chain would fold, in which case only a quantum computer can be used to simulate it. These many variables, taking as an example the ones you mentioned, the atomic forces, are somehow grouped in order to facilitate the calculations; thus, it is not ...

1

I will give you a few scenarios where matrix factorisation stills works pretty well. Topic Modelling : Given a matrix of Document as Rows and Terms/Words as column you can use Non Negative Matrix factorisation to identify Topics. Number of Topics is defined by user or can be treated as hyperparameter. Image Ref : https://towardsdatascience.com/nmf-a-visual-...

0

When we are training deep neural Network gradient tells how to update each parameter, under the assumption other layers do not change.In Practice, we update all the layers simultaneously. When we update, unexpected results can happen because many functions composed together are changed simultaneously using updates that were computed under the assumption that ...

1

Common loss functions, like the cross-entropy or mean squared error, are chosen because, if you minimize them, you are actually maximizing the likelihood of the parameters given the observed data. In other words, you are trying to find the probability distribution $p(y \mid x; \theta)$, parametrized by $\theta$ (the parameters of your model), which is most ...

1

As DKDK said, Indeed one could fit both linear and exponential function and see which one has smaller residual, without using any complex AI. But OTOH this could be a great toy-problem for learning about neural networks. You could have a network with these parts: A network with a final sigmoid activation, which predicts whether the function is linear or not....

2

How about dividing the problem? You can first train a classification model that predicts the type of function (linear or exponential). Then you can use your seperately trained nn depending on the classification output. P.S. I'm not sure why you would use a neural network for this problem. Fitting a linear/exponential function seems to be a relatively simple ...

0

Mixture of Experts might be what you are looking for. A Mixture of Experts model (MoE), divides a task into subtasks and designs seperate models for each of the tasks (This would be N in your case). It also defines a gating model to decide which expert to use, and during inference it uses the gating model output to pool/select predictions and makes the final ...

0

As you made this experiment available on Colab, I was able to test my thoughts on it, which was handy. First, the simple "fix" is to run many epochs. Eventually even your relatively small and simple network will learn to predict whether the mean of a large randomly-generated vector is greater than the expected value of all the elements. In my tests,...

0

I think, that the answer depends on the actual setting and parameters of the problem. Lookup table One needs to store the whole lookup table in the memory - say N points. The number of operations to extract elements depends on the way values are stored in the memory. In order to get a single value given the input $x$, you need to perform $O(\log N)$ ...

0

Convolutional Neural Networks are mostly used for all kind of computer vision tasks. Here you can find a tutorial on how to train a CNN for image classification from scratch.

2

Optimizing the cross-entropy is equivalent to optimizing the log-likelihood of the parameters given the data, $\ell(\theta)$, which is what we want, i.e. find the parameters that most likely generated the data. So, the likelihood is defined as $$\mathcal{L}(\theta) = P(y \mid x; \theta),$$ i.e. a function of the parameters $\theta$. The log-likelihood is ...

2

The background being an unbalance class is a well known problem in image segmentation. Before digging into custom losses you should take a look to existing ones that address this specific issue like the Dice Loss or Focal Loss, the latter being more tunable having a extra hyper parameter that can be optimized. You can easily find on github tensorflow ...

Top 50 recent answers are included