Questions tagged [deep-learning]

For questions related to deep learning, which refers to a subset of machine learning methods based on artificial neural networks (ANNs) with multiple hidden layers. The adjective deep thus refers to the number of layers of the ANNs. The expression deep learning was apparently introduced (although not in the context of machine learning or ANNs) in 1986 by Rina Dechter in the paper "Learning while searching in constraint-satisfaction-problems".

Filter by
Sorted by
Tagged with
1
vote
0answers
31 views

Why does research on faster Transformers focus on the query-key product?

A lot of recent research on Transformers has been devoted to reducing the cost of the self-attention mechanism: $$\text{softmax}\left(\frac{Q K^T}{\sqrt{d}} \right)V,$$ As I understand it, the runtime,...
0
votes
0answers
89 views

Understanding gumbel-softmax backpropagation in Wav2Vec papers

I'm studying the series of Wav2Vec papers, in particular, the vq-wav2vec and wav2vec 2.0, and have a problem understanding some details about the quantization procedure. The broader context is this: ...
1
vote
1answer
36 views

Can I use the transformers for the prediction of historical data?

Can I use the transformers for the prediction of wind power with the historical data? Dataset Datetime, Ambient temperature (Degree), Dewpoint (Degree), Relative Humidity\n (%), Air Pressure, Wind ...
2
votes
3answers
128 views

Should I allow NN to infer relationships of inputs?

This question is assuming a sequential, deep neural network Given some features [X1, X2, ... Xn], I'm trying to predict some value Y. The raw data available to me contains feature ...
0
votes
0answers
24 views

What can be recommended reputed deep learning journal(s) or conference(s) that do accept architectures in comparatively less time?

In the domain of deep learning, architectures are highly important. Many research journals and conferences do accept the proposed architectures as the contribution. Since the domain of deep learning ...
2
votes
1answer
45 views

Is there any reason behind bias towards max pooling over avg pooling?

Consider the following excerpt taken from the chapter named Using convolutions to generalize from the textbook titled Deep Learning with PyTorch by Eli Stevens et al. Downsampling could in principle ...
1
vote
0answers
19 views

What are the defining moments that make community realise the potential of deep learning?

Consider the following paragraph from the chapter named pre-trained models from the textbook titled Deep Learning with PyTorch by Eli Stevens et al. The AlexNet architecture won the 2012 ILSVRC by a ...
1
vote
2answers
48 views

When would it make sense to perform a gradient descent step for each term of a loss function with multiple terms?

I am training a neural network using a mini-batch gradient descent algorithm. Now, consider the following loss function, which is composed of 2 terms. $$L = L_{\text{MSE}} + L_{\text{regularization}} \...
0
votes
0answers
48 views

What is the effect of gradient clipping by norm on the performance of a model?

It is recommended to apply gradient clipping by normalization in case of exploding gradients. The following quote is taken from here answer One way to assure it is exploding gradients is if the loss ...
0
votes
0answers
23 views

Is the multi-head attention in the transformer a weighted adjacency matrix?

Are multi-head attention matrices weighted adjacency matrices? The job of the multi-head-attention mechanism in transformer models is to determine how likely a word is to appear after another word. In ...
0
votes
0answers
24 views

Without using data augmentation gives results better than using data augmentation

I am a beginner to deep learning, I'm doing the image classification problem on a small self plant disease imaging dataset (400 images). I am doing transfer learning (pre-trained ...
1
vote
1answer
36 views

What is the best open source python repo for facial recognition? [closed]

I am looking for best open source python repo for facial recognition. Best if it uses tensorflow backend. I know you can train images to recognize. Yolo can be used if trained on face. To name the ...
1
vote
1answer
176 views

What exactly happens in gradient clipping by norm?

Consider the following description regarding gradient clipping in PyTorch ...
0
votes
0answers
12 views

How sensitive are LSTM's to random zero values in its target feature when training?

I have worked with lstm's in the past, specifically for time series forecasting. However, the target feature in these time series were relatively "stable". With the loosely defined "...
0
votes
1answer
41 views

When can we call a feature "hierarchical"?

Features in machine learning are the attributes of the elements of a data set. They are considered as random variables in probability. Consider the following excerpt from 1.1: The deep learning ...
2
votes
1answer
41 views

What can be an example for the prior knowledge used in Deep Learning systems?

It is known that machine learning algorithms expect feature engineering as an initial step. Now, consider the following paragraph, taken from 1.1 The deep learning revolution of the textbook named ...
1
vote
0answers
29 views

How should I choose a reinforcement learning algorithm? [closed]

I'm starting a new RL project. I'm familiar with Deep Q-Learning because of an old project where I used it, but I'm not sure I chose correctly back then. Why should or shouldn't I choose DQN, or any ...
1
vote
0answers
26 views

How do you decide that you have tested enough hyper-parameter combinations for a specific neural network architecture?

How do you decide that you have tested enough hyper-parameter combinations for a specific neural network architecture to discard it and move on to a new model? Do you have a structured (generic) ...
1
vote
0answers
20 views

Monocular depth estimation

I am currently reading the paper towards robust monocular depth estimation and I have 2 doubts about it. First of all the paper stated that there are 2 types of depth annotated, dense and sparse. What ...
1
vote
1answer
42 views

Do you need a terminal state when using double deep q networks?

I just got my agent training, and I'm wondering if the terminal flags are necessary when sampling from the replay buffer. The game I'm implementing the agent in has two different ways the game can end,...
0
votes
0answers
27 views

What can cause massive instability in validation loss?

I'm working with very weird data that is apparently very hard to fit. And I've noticed a very strange phenomenon where it can go from roughly 0.0176 validation MSE to 1534863.6250 validation MSE in ...
2
votes
1answer
44 views

Does a bigger neural network learn "worse" representations than a small neural network when the amount of data isn't enough?

Assume we have a neural network and we want to train it on a classification problem. The hidden layers of the neural network are kind of feature representations of the input data. If the neural ...
1
vote
1answer
44 views

Are there any animation tools available to visualise and simulate deep neural networks? [closed]

Deep learning researchers have to work with a lot of models. The models may include different types of Layers: They include convolutional neural network layers, recurrent neural network layers, batch ...
0
votes
0answers
26 views

Training a neural network using several data sources with quality flags

I have been searching for a specific problem in training a NN and hope someone is able and willing to help as I cannot find a solution. The problems is that I have a spatial data set with 3 sources of ...
0
votes
2answers
38 views

In mini-batch gradient descent, do we pass each input in the batch individually or all inputs at the same time through the layer?

In the stochastic gradient descent algorithm, the weight update happens for every training sample. In the mini-batch gradient descent algorithm, the weight update happens for every batch of training ...
0
votes
2answers
61 views

Is there any relationship between the batch size and the number of epochs?

I am currently running a program with a batch size of 17 instead of batch size 32. The benchmark results are obtained at a batch size of 32 with the number of epochs 700. Now I am running with batch ...
2
votes
2answers
67 views

How general is generalization?

I am sorry but I have to explain my question using an example, I do not know how to ask it in proper scientific terms. Let's assume, I have trained a deep learning model on classifying hand gestures, ...
0
votes
0answers
15 views

Brain tumour detection using CNN

I have a fairly basic mathematical and implementational understanding of ML algorithms and CNNs, and I am trying to think of an approach for this task: https://www.kaggle.com/c/rsna-miccai-brain-tumor-...
0
votes
0answers
20 views

How to assess the goodness of a text generation algorithm

Take a RNN network fed with Shakespeare and generating Shakespeare-like text. Once a model seems mathematically fine, as can be assessed by observing its loss and accuracy over training epochs, how ...
0
votes
0answers
13 views

Counting number of coaches in a train from real time video feed

I have a real time video feed of a train platform. I was able to detect coaches using CNN based model. But how can I calculate number of coaches in the train that passed the platform as well as the ...
0
votes
0answers
34 views

Data Augmentation for Object Detection - Polygon Region Shape

I'm looking to run a Mask RCNN code on my dataset of about 2700 images. The images are too large and I would like to resize them, and I would also like to add some shear, scale and zoom augmentations. ...
0
votes
0answers
26 views

What all are the known reasons for the decline in the performance of a neural network if we keep on increasing the depth of it?

Progress in many application tasks in artificial intelligence is achieved by increasing the depth of the neural networks. But if we keep on increasing the number of layers in the neural network, the ...
2
votes
1answer
227 views

Is "width of a neural network" a wrong phrase?

Depth of the neural network is equal to the total number of layers in the neural network except input layer. so, neural network with more number of layers are called deep neural networks. Width, in ...
0
votes
0answers
18 views

multi agent deep deterministic policy gradient for discrete actions

I am solving a multi agent problem where each agent has a critic and actor. The problem I am solving has discrete actions and discrete states. I came cross multi-agent deep deterministic policy ...
1
vote
0answers
32 views

What are Linear and Non-Linear Features of an image in the context of Convolutional Neural Network?

What features of image are linear or non-linear, any example ?
4
votes
2answers
62 views

Should I repeat lengthy deep learning experiments to average results ? How to decide how many times to repeat?

I am doing my MSc thesis on deep learning. My model takes many hours to train. Part of what I do is trying different parameters and settings hoping that they will achieve different results. But I ...
2
votes
1answer
42 views

Where can I read about upsampling methods in detail?

In deep learning, we encounter the upsample blocks several times, especially when we deal with images. Consider the following statements from description regarding UPSAMPLE in PyTorch The algorithms ...
1
vote
0answers
10 views

Continue teaching pre-trained network without forgetting previous data set

I have a rather interesting problem here; I work in the field of image classification for quality assurance. For this I have a dataset of about 1 million images, which I have used to train different ...
1
vote
0answers
32 views

Machine learning with raw data alone / or raw data with its statistics

My question is very general and it does not originate from a specific problem. Let's assume that, through experience, we have learned that some statistical property of a set of data is important in ...
0
votes
1answer
124 views

What is better to use: early stopping, model checkpoint or both?

I want to get a model which works best, what should I go for while training the model, ModelCheckpoint, EarlyStopping, or both?
0
votes
0answers
37 views

How to increase accuracy of image orientation classification (Left, Right, Center)?

I am working on classifying images in "Left", "Right", "Center", "Back". Training and Validation images look like this: The images are "Left", "...
3
votes
1answer
71 views

Why should one ever use ReLU instead of PReLU?

To me, it seems that PReLU is strictly better than ReLU. It does not have the dying ReLU problem, it allows negative values and it has trainable parameters (which are computationally negligible to ...
0
votes
0answers
31 views

Is there anything wrong with this YOLO loss function?

I have implemented the YOLOv1 loss function as: ...
0
votes
1answer
57 views

Is it normal that the values of the LogSoftmax function are very large negative numbers? [closed]

I have trained a classification network with PyTorch lightning where my training step looks like below: ...
1
vote
1answer
65 views

Multi Agent Deep Reinforcement Learning for continuous and discrete action

I am looking to have a cooperative multi agent reinforcement learning framework where one agent has a discrete action space and another agent has a continuous action space. Is there a way to do this ...
4
votes
1answer
86 views

How to approach a blackjack-like card game with the possibility of cards being counted?

Consider a single-player card game which shares many characteristics to "unprofessional" (not being played in casino, refer point 2) Blackjack, i.e.: You're playing against a dealer with ...
0
votes
0answers
70 views

What's wrong with my understanding of how RNNs work?

Recently, I've been trying to derive the mathematics behind various Neural Network structures. I managed to derive the MLP and tested it to be on par with a Keras implementation (Using the MNIST ...
1
vote
1answer
69 views

Different ways to calculate backpropagation derivatives, any difference?

I'm studying error backpropagation in neural networks. I am interested in why we use only one path on the computational graph to get the value of the derivative for a weight? I ask the question ...
1
vote
0answers
51 views

What are strategies for data driven weights initialization?

I am beginner in deep learning and currently training a few neural networks (Pytorch) for problems in audio and speech. For my tasks, simple feed-forward networks are working well enough. I use basic ...
0
votes
0answers
38 views

What are mathematically the factors of variation in deep learning?

The following paragraph from an answer tells us about factors of variation Factors of variation are some factors which determine varieties in observed data. If that factors change, the behaviour of ...

1 2
3
4 5
33