New answers tagged

0

The answer above is very concise but I will try to give an ELI5 example. I also agree with @nbro that attention does not exclusively mean transformer architecture. Before attention What is the height of the youngest female child of the father of your mother's first cousin? That query is convoluted, depends on your good memory of your family tree's ...


1

After a long wait and some digging, I accidently found what I was looking for. In 2015, polish researcher Dominika Traczyk publish an article presenting CERMINE, a solution for the posted problem. His solution is SVM-based, but the article gives good insights for an alternate Neural Network version. The article is open access and can be found on the Springer ...


2

Let's start by stressing out that in the literature unfortunately the term attention is still used widely without any precise consensus around the technical details, the only constant across papers is that attention should be used when a model is capable of learning, or focusing on local vs global patterns in the data we use for training. And with "...


1

Transformers, being a general-purpose sequence model can be used for Time-Series forecasting. There are some papers dedicated to the use of Transformer for time-series prediction and blogs. The main ingredient for the autoregression in predictions is the mask in Transformer encoder. When the next element is predicted, tokens in the sequence attend only to ...


6

Unfortunately, even with large amounts of training data, hyperparameter choices can strongly influence the performance of a trained model. What you can usually drop when you have large amounts of training data is regularisation. If your training examples cover the function space you are learning really well, then it is harder to overfit the training data. ...


3

You don't NEED a hyperparameter tuner, but it can help in various situations. For example, if your model is not training well, perhaps using a tuner can help. It's hard to say in which hyperparameters you would be turning over in your specific model, but for some specific hyperparameters if you choose a bad value your model won't learn or diverge. Take for ...


0

Closed Loop Network Step-Ahead Prediction Network The function CLOSELOOP replaces the feedback input with a direct connection from the output layer. Step-Ahead Prediction Network also known as removedelay function helps to remove delay to neural network’s response In Closed loop networks, its own predictions become the feedback inputs. targets with a delay ...


1

Try removing the dropout before the prediction layer. I couldn't find the paper or article I read about this (will update the post once I find it), just found a Cross Validated post which does not add much information. As you are If you are lowering the learning rate, you should also lower the batch size accordingly. As for Batch Normalization layers, they ...


1

What is the No Information Rate (NIR)? I.e. what are the percentages of positive and negative labels? Have you looked at the predictions of your model? If it's all 0's or all 1's then it probably learned nothing, other than predicting the majority class. When it comes to architectural choices and hyperparameters, especially if you start working with NNs, ...


0

It seems your problem is more related to Face Identification than Face Recognition. I understand you are looking for the implementation using a NN based approach, but if you're open to giving it a try to other approaches you could consider using Eigenfaces, which is based in PCA. For that, you can find some references and code implementations. Datasets you ...


2

Does the ANN's training data include the proper output for every neuron? The short answer is: no (not usually or directly). The long answer is that you can train neural networks in different ways. There's supervised, unsupervised, reinforcement learning/training, or even other ways (e.g. online learning). The most common way of training neural networks is ...


1

The most generic answer to this question is: the same metrics you use to evaluate the quality of your model during training or in test phase. (Plus the timing of inference if you're referring to computational efficiency) And I'm not referring to any specific metric yet cause that's really task dependent. But in general if you have a model that perform a task ...


0

It might be a good idea to normalized the coordinates with their counterparts in all 4 objects: for example, if you use min-max scaler, you should scale x1, x2, x3, x4 together, and the same for y and z coordinates. This is assuming your coordinate system is infinite. If you have a finite coordinate system, or there exists some natural limits to the maximum ...


0

You could use Mutual Information between the model's prediction, and that particular feature as a regularization term. This will minimize the dependence of the output to that particular feature. Note that simply removing the feature from the dataset might not work if other features are associated with the feature which you don't want your model to depend on.


2

They have a few similarities, but they are quite different. Let me first give you a general description of both approaches/algorithms, so that you start to get a sense of their differences and similarities. Description Gradient descent (GD) can be applied to solving any optimization problem where your loss (aka cost or objective) function is differentiable ...


1

I am not sure if there really are contradicting opinions on this matter. CNNs, RNNs, LSTMs all have specific types of data they are good at predicting. Depth and width, or in general the size of the neural network mostly depends on the size of your dataset. You don't want to build a too large network that will overfit the available data, which can usually be ...


0

Yes, it is a well known problem called Curse of Dimensionality. It happens when a finite number of data samples is used to train a network with a high-dimensional feature space (very deep network). With regard to your question: yes, smaller networks (representational spaces with lower dimensions) describe better smaller datasets. Why? Because of data ...


1

I think there's a crucial point missed in the question, touched by jros answer but without further elaboration. If you train a model on domain A: single lightning condition and test it on domain B: two lightning condition then you're not evaluating generalization but transfer learning capabilities. Or to phrase it differently you're evaluating how close ...


0

This depends on the behaviour you want. If the ambiguous sample's ground truth is classified by a range of people, your network will get an average* based on that group. If it's only by one person, your network will be biased to how that one person classifies these samples. Alternatively, depending on your loss function, you could train the network to ...


0

Generalization In machine learning, generalization describes a model's ability to properly correct its algorithms to predict new data from the same distribution as the data used to train the model. By providing additional training for your model (on data with varying lighting conditions), you are correct that you would be increasing the capabilities of your ...


0

How many optimisation iterations are performed on a mini-batch? Just one, as you suspected. then how does an optimisation algorithm like adam work which uses past gradient information? It uses the gradient estimates from each mini-batch as its input sequence. It seems strange since then gradients from past mini-batches are being used to minimise the loss ...


1

In the usual scenario, case 2 occurs. In the deep learning frameworks, Tensors have special dimension (usually corresponding to the 0 axis) which numerates the example in the batch. Look for example in the PyTorch documentation of Conv2d or Tensorflow documentation of Conv2d. The same is true for any Layer - Linear, MultiheadAttention, RNN. All samples from ...


0

What happens in mini-batches is not very different from the way updates are made in batch gradient descent, only the number of samples is different. In mini-batch, you process all the data in the batch, and the update happens after that. It is detailed in this video after 6:11.


0

The smaller the batch_size is, the larger number of batches is processed per one epoch. On one hand, since one makes more steps per epoch, one can think, that less epochs are required to achieve the same level of accuracy. On the other side, smaller batch size leads to more noisy and stochastic estimates of the gradient, therefore, convergence would not be ...


0

Smaller batch size means the model is updated more often. So, it takes longer to complete each epoch. Also, if the batch size is too small, each update is done without "seeing" all the data - the batch itself might not be a good representative of the dataset. So, there might be too much "wiggling", which makes it harder to get real ...


2

This width of a neural network *layer is an agreed upon term. *The width of a neural network is generally the width of the widest layer of the neural network. *I would caution how you use the phrase "width of a neural network" due to interpretability and scale, *and the fact that neural networks often contain layers with varying numbers of neurons, ...


2

To get a full understanding of your problem, one would like to know what approximately the $n$-features are. Whether, it is about the geometrical structure, protein is described by a graph, where vertices correspond to atoms and edges to bonds within them - I would consider use of GraphNN, there is some research, that has demonstrated the success of GraphNN ...


1

This should be possible but I've never seen it done in practice. Whether or not this will even actually work is unclear to me and will be highly dependent both on your training data and choice of loss. I'd take a step back and look into the literature to see if you can't find a more established approach to your problem, perhaps with RNNs. That being said, I ...


Top 50 recent answers are included