New answers tagged deep-learning
0
In general, yes.
Stacking more layers and adding non-linearities will form a better function approximation (neural nets are basically function approximators), and when trained with the current regularization for each layer (such as L2 or L1) will cause your model to learn a better mapping, and hence generalize better.
If you don't regularize, it will overfit....
1
In your case the most probable explanation would be the case of overfitting. The model with too many hidden layers have lots of parameters. By means of all these parameters the model is remembering stuff from the training data itself instead of generalizing by learning the useful patterns.
As a rule of thumb if you increase the number of hidden layers more ...
0
Here are a couple of Kaggle Kernels, Notebooks and Tutorials for Image Captioning.
Kaggle Kernel | Neural Image Captioning: 🌄 -> 💬
Kaggle Kernel | Show Attend and Tell
Kaggle Kernel | Flickr Image Captioning : TPU, TF2 & Glove
Tensorflow Tutorial | Image captioning with visual attention
Show and Tell: A Neural Image Caption Generator by Vinyals ...
1
You are talking about model parallelism. But, that's not the reason RNNs/LSTMs are not in vogue.
Imagine your ability to read the first line of a page and going on reading and still making connections to the first line until the end of the page.
Can RNNs/LSTMs do that? No.
Can Attention (i.e. Transformers) do it? Yes.
The reason is simple Attention is ...
0
Since it is a trained network already, when you run an example through it, the gradient will not have a very high variance.
The gradient varies a lot when you are training a network from the scratch but then it stops varying much since it understands the pattern.
1
That is exactly a neural network works like.
Suppose you have a 1000 examples. How you train a network is: First, you divide these 1000 into maybe 100 batches (10 each). After that's done, you feed a batch to the network get its output and compare it with the ground truth, whatever is the error gets backpropagated. Then, for the next batch and then another. ...
2
Can residual connections be beneficial when we have a small training dataset?
The usual rule of data science investigations applies here: Try it, measure the results, then you will know.
It is very hard to tell, a priori, whether a specific architectural or hyperparameter choice will impact the performance of a neural network on a given problem.
In this ...
1
Yes, it is possible. What you have shown in case of ANN is what happens in a regression model using NNs. What you have shown in case of RNN is what happens when you are doing sequence-to-sequence translation (like French to English).
If you want to get single values like in case of ANN, suppose you are doing regression, then, in the end, you will flatten the ...
0
To speak to your question about how Chinese to English translation can be a computation, it first requires a way to turn the base units of translation (tokens) into something computable. One basic way is to define the set of your vocabulary terms and create a gigantic matrix (typically called an embedding) with each column representing a token as well as one-...
2
A function is simply a procedure that maps a particular input to a particular output. You put in $X$, and the function computes $Y$. Those $X$ and $Y$ can take many different forms. It could be mapping one number to another number (convert miles to kilometres), mapping sound to text (name that tune), mapping text to text (translate languages), mapping a ...
0
Ok, I solved this problem
The simple thing was that learning rate was too big
I changed the code to this
LR = batch_size/((z+1)*100000)
LR=LR/3
instead of
LR = batch_size/((z+1)*1000)
LR=LR/3
and it seems to work well
1
One way to look at intelligence is it's the way to compress the universe. That means we have a short mental representation of meaningful concepts.
For example, if I would say "there is a red swan in your building, it's dangerous and can kill you", you already have concepts of "red", "swan", "danger" and this easy ...
1
That equation is just an assumption that we make about the relationship between a response variable (aka dependent variable) $y$ and a predictor (aka independent variable) $x$, i.e. the response variable (target) is an unknown function $f$ of the predictor $x$ plus some noise $\epsilon$ due to e.g. measurement errors (caused e.g. by damaged sensors). So, if ...
3
Not necessarily. The neural network (or whatever else you use) is a model of what you are trying to do, and usually models are not able to perfectly model reality, as it is too complex. A noise term is generally used to represent that, ie the imperfection of the model's relationship with the actual world.
0
For a regressor, it can work fine to have an output layer that is linear.
The composition of two linear functions is also linear, so in a deep neural net, if all layers are linear it can only learn a linear function. As Daniel B explains, XOR is a good example of a function with no useful linear approximation.
0
A simpler answer is that for a standard neural net, the asymptotic behaviour is the asymptotic behaviour of the output neurons. For example, if the output layer is ReLUs, then the asymptotic behaviour is necessarily linear.
In your case, since you want it to be asymptotically constant, you can use the slightly old-fashioned choice of sigmoid units in the ...
1
For standard NNs, their extrapolation behavior an important aspect for financial applications cannot be controlled due to complex functional forms typically involved.
Neural Networks with Asymptotics Control discuss how they overcome this significant limitation and develop a new type of neural networks that incorporate large-value asymptotics, when known, ...
0
The short answer is yes.
When you merge the test set into the train set, you try to squeeze available data till the last drop. The cons and pros of this approach have been considered in other questions in the network 1, 2. But if you decided to go for it, there is no point to not use the whole dataset for the scaling transformation, as the trend "more ...
0
Each machine learning model should be trained by constant input image shape, the bigger shape the more information that the model can extract but it also needs a heavier model.
A model's parameters will adapt with the datasets it learns on, which means it will perform well with the input shape that it learned. Therefore, to answer your question "What I ...
0
In all pre-trained models, the input image has to be the same shape; the transform object resizes the image when you add it as a parameter to your dataset object
transform = T.Compose([T.Resize(256), T.CenterCrop(224), T.ToTensor()])
dataset = datasets.ImageNet(".", split="train", transform=transform)
T.Resize(256) changes the image ...
1
Okay - the answer is here https://explained.ai/matrix-calculus/#sec6.2 and it is pretty involved. Basically, there is a difference when you derive the equation for one neuron and when you have to do practically for a set of neurons. The answer is matrix calculus. Here goes from what I could make out. Feel free to correct if I am wrong
Gradient Vector/Matrix/...
1
I'll try to answer on more general questions
Is it ok that model performs better on validation, then on train?
It's certainly fine if you use techniques like dropout or data augmentation and the difference is not that big. Because in case of dropout for train you use part of the network, and for validation the whole.
I'm suspicious my model is too good. ...
0
I'm new to all this so take what I say with a grain of salt and not as fact, I don't have any formal education or training. I believe when you're referring to inversion predictions, you're not overthinking you're underthinking. For anything to have value it must also have an inverse or else there's no way to cognitively perceive it (contrast) otherwise you'...
0
There is an implementation of the paper ("Adversarial Sparse Transformer for Time Series Forecasting"), in Python using Pytorch, here. Although it has the training and evaluation functionality implemented, it appears to be lacking a function for running a prediction. Maybe you can fork it and extend it.
UPDATE
There is also a paper, "Informer:...
2
The short answer is no, you shouldn't do that.
There is a "distribution shift" thing when you have different x-y relation on the validation set then on the train set. The distribution shift would deteriorate your model performance and you should try to avoid that. The reason it's bad - ok, you find the way to fix the model for validation data, but ...
0
We basically distinguish between 3 forms of batch training:
$$Loss_{minibatch} = \sum_{m} l_m(\mathbf{W},t_m) \;\;\; with \;m \;\epsilon \; M$$
where M is a (random) subset of the whole dataset.
$$Loss_{batch} = \sum_{b} l_m(\mathbf{W},t_b) \;\;\; with \;b \;\epsilon \; B$$
where B is the whole dataset.
$$Loss_{stochastic} = l_i(\mathbf{W},t_i) $$
where i is ...
1
Okay, I think it's better if we distinguish loss and accuracy first via Jeremy's answer, and I agree with him with the sentence "low or huge loss is a subjective metric".
The loss value is easy to affect by noise from data and significant increase with a few error data points. My advice in this case is to use more evaluation metrics, and understand ...
0
LASER creates multilingual contextualized word embeddings, what you do with them is up to you. You can use this as a feature extraction and add whatever you want to the end of the network.
I believe the implementation by facebook does not let you change the weights of the LASER model itself, they are froozen to the best of my knowledge.
So, yes, you can use ...
2
generally the approach is to have a separate head. For example, imagine you have latent vector $z_k$, you would output two values: $h(z_k)$ and $f(z_k)$ where $0 \leq h \leq 1$ and $b_0 \leq f \leq b_1$ where $b_0$ and $b_1$ are your bounds.
In thios setup, during inference you would check $h_k$ and if its greater than some threshold (usually .5), youd ...
0
After working on it for a while this is what I got.
Concerning proposition 1 in the paper, a rigorous statement could be the following version of the Gradient Theorem for line integrals:
Proposition 1. (Gradient Theorem for Lipschitz Continuous Functions).
Let $U$ be an open subset of $\mathbb{R}^n$. If
$F : U \to \mathbb{R}$ is Lipschitz continuous, and
$\...
1
What you want to do is called multi-task learning. Here's what you do:
Create a second Input.
Attach it to 1D CNN (2-3 layers), so it aggregates this tabular information.
Concatenate this feature with the intermediate feature generated by the U-Net using Concatenate layer.
Put a dense layer of 2 after this.
Put softmax with units = number of classes.
Add CE ...
1
The lower bound in MINE is as follows:
$$\widehat{I(X;Z)}_n = \sup_{\theta\in\Theta} \mathbb{E}_{\mathbb{P}_{XZ}^{(n)}}[T_\theta] - \log{\mathbb{E}_{\mathbb{P}_X^{(n)} \otimes \hat{\mathbb{P}}_Z^{(n)}}[e^{T_\theta}]}$$
Here $\mathbb{\hat{P}^{(n)}}$ denotes the empirical distribution that we get from n i.i.d samples of $\mathbb{P}.$
Note that in the above ...
0
Tl;dr max-pool
You can see in the diagram, everywhere there are a variable number of inputs (pickups, units, hero modifiers/abilities/items), a max-pool follows, though I don't know the specifics of the max-pool implementation.
From https://neuro.cs.ut.ee/the-use-of-embeddings-in-openai-five :
Notice that while the number of modifiers, abilities and items ...
2
Yes, neural networks learn features themselves freeing you from the need to manually engineer them. I will illustrate it here with a toy problem.
Let's assume that we want to learn the areas of parallelograms built on pairs of vectors:
The input data are six coordinates: $(x_1, y_1, x_2, y_2, x_3, y_3)$.
import numpy as np
n_tr = 1000 # training data
x_tr = ...
2
Feature engineering may be necessary when one cannot achieve acceptable error rate — within a budget or in principle.
NN may be stalling due to information bottleneck: too many pigeons, not enough holes. In that case, custom features may provide slightly better information compression. (Alas, this is not a panacea: some layer(s) may still be too narrow. That'...
Top 50 recent answers are included
