Questions tagged [deep-neural-networks]
For questions related to deep neural networks, which are artificial neural networks with "many" layers, where "many" can vary depending on the context.
201
questions
0
votes
0
answers
25
views
Longer DNN training times when using evolutionary algorithms
I am comparing my deep neural network (DNN) performance when using 2 types of optimizers: gradient-based Adam (properly tuned) and a population-based optimization algorithm (e.g., genetic algorithm (...
2
votes
1
answer
87
views
Can models like chatGPT learn functions with infinite domain or range
Lets assume two types of prompt:
A fixed prompt for which reasonable responses can be infinite. For example:
> output a random number
> output a palindrome
...
- 941
2
votes
1
answer
662
views
Can we use a pre trained Encoder (BERT, XLM ) with a Decoder (GPT, Transformer-XL) to build a Chatbot instead of Language Translation?
I was wondering if the BERT or T5 models can do the task of generating sentences in English. Most of the models I have mentioned ...
CommunityBot
- 1
3
votes
0
answers
26
views
Why does training converges when the norm of gradient increases?
This is from deep learning book by Ian Goodfellow and Yoshua Bengio and Aaron Courville.
When training converges well, I thought the gradient should be at local minima. But the book says it often does ...
- 205
1
vote
1
answer
125
views
Formally, what are the layers in an Artificial Neural Network?
You may not believe it, but I am an ANN expert. Perhaps, for that reason, I am unable to grasp completely what the layers are in a Deep Forward Artificial Neural Network (DFANN).
According to the Deep ...
- 11
0
votes
0
answers
38
views
Does the number of epochs measure a correlation?
i have built a two-layers neural network (1000 => 1000) to predict a dynamical system driven by two real-world parameters.
When using the first parameter as input to the first layer, training the ...
- 11
0
votes
1
answer
612
views
Is it redunant to add more layers to a neural network with same number of neurons as the previous layer?
Lets say I have a neural network with three layers and the last layer has 3 outputs.
If I added additional layer of 3 neurons to the end of the network, would that be a more powerful neural network?
...
- 2,890
1
vote
1
answer
502
views
Why MSE and MAE yield poor results when used with gradient-based optimization for classification?
Deep learning book chapter 6: In 6.2.1.2 last paragraph:
Unfortunately, mean squared error and mean absolute error often lead to poor results when used with gradient-based optimization. Some output ...
- 942
0
votes
3
answers
724
views
why validation accuracy be greater than training accuracy for deep learning models? [closed]
I hope you are well.
I had a problem and didn't understand the answers given on questions similar to my question.
If possible, please answer this problem in a simpler way.
Val_acc : %99.4 _
Train_acc :...
- 2,524
1
vote
1
answer
158
views
MobileNetV2 - Some particularities
So I was studying MobileNetV2 architecture and came across this table from the original paper that represents its architecture:
Table Description: "Table 2: MobileNetV2 : Each line describes a ...
- 356
0
votes
1
answer
113
views
Training a neural network in full batch training
It is a trend in deep learning to train models using multi-batches, i.e., to show the model a subset of the entire dataset for each weight update. In some cases, as in continual learning, we see that ...
- 5,298
1
vote
4
answers
159
views
Why, in deep learning, do we get computational power by going deeper?
I know by the expressiveness of a neural networks that it can be seen as a chain of function compositions, i.e. $g(f(.. z(x)..))$ and also that, if we go deep, we can approximate complex functions $f: ...
- 11
1
vote
0
answers
39
views
Why "Good Model" that performs great on holdout validation data fails on production data
I have this binary regression model that has ~500 futures with an unbalanced dataset with the following results.
...
- 23
1
vote
1
answer
235
views
How to handle out-of-bound values in Production data?
So I have this model but the data may vary. And it is virtually impossible to always have the values in bounds. If I do I`d have to use larger period leading to concept shift which is worse.
The ...
- 5,298
1
vote
0
answers
87
views
How can I get an integer as output for continuous action space PPO reinforcement learning?
I have a huge discrete action space, the learning stability is not good. I'd like to move to continuous action space but the only output for my task can be a positive integer (let's say in the range 0 ...
- 111
3
votes
1
answer
292
views
Is batch normalization not suitable for non-gaussian input?
I generate some non-Gaussian data, and use two kinds of DNN models, one with BN and the other without BN.
I find that the model DNN with BN can't predict well.
The codes is shown as follow:
<...
CommunityBot
- 1
1
vote
1
answer
63
views
Do you need single or multiple networks to detect multiple faces?
Given pictures with multiple features such as faces, can a single AI algorithm detect all of them, or for better reliability is it preferred to use separate instances?
In other words, I'm talking ...
- 3,840
1
vote
1
answer
39
views
How to identify important features in data?
I have a couple opportunities to write a paper, or papers over some of the neural networks I have made.
I was wondering if there are anyways to figure out why the neural network classifies the data I ...
- 11
1
vote
0
answers
27
views
How Can We Create Neural Networks with Different Depths and Widths But Same Number of Parameters?
Right now I am doing a research project investigating how the depth of a Neural Network affects its capacity to learn. In order to do this, I wanted to test different Networks with the same number of ...
- 11
0
votes
1
answer
255
views
How to compare memory requirements for tabular Q-learning vs deep neural network?
I want to compare the space complexity/memory requirement of tabular Q-learning v.s. deep neural Q-network (DQN). I think DQN would be faster and Q-table has a disadvantage at large table sizes but ...
- 32.1k
1
vote
1
answer
44
views
How to preserve Markov Property in Deep Reinforcement Learning when using "mixup" or "mixreg"?
I've read through these two papers:
(original about "mixup") https://arxiv.org/pdf/1710.09412.pdf
(variant for RL, "mixreg") https://arxiv.org/pdf/2010.10814.pdf
They are about a ...
- 491
2
votes
1
answer
739
views
Should I use an unsupervised approach or train a classifier with many classes to build a deep image feature extractor?
I'd like to build a deep feature extractor of images (using a Bi-linear CNN).
What would lead to the best results:
an unsupervised approach (such as https://iopscience.iop.org/article/10.1088/1742-...
- 40.5k
2
votes
0
answers
1k
views
How many layers do GPT-3, AlphaFold 2, and DALL-E 2 have?
Unsuccessfully, I tried to find out the "depth" (definition below) in large neural networks such as GPT-3, AlphaFold 2, and DALL-E 2.
Formally, my question is about their computational graph:...
- 40.5k
1
vote
0
answers
244
views
How to define a custom layer in Pytorch [closed]
I am new to PyTorch and seeking your help regarding a problem I have. I need to add a costume layer to a NN in training phase. Please see the figure which shows a simple DNN with the custom layer. NN ...
- 11
2
votes
1
answer
35
views
Why does backprop algorithm store the inputs to the non-linearity of the hidden layers?
I have been reading the Deep Learning book by Ian Goodfellow and it mentions in Section 6.5.7 that
The main memory cost of the algorithm is that we need to store the input to the nonlinearity of the ...
- 903
-1
votes
1
answer
513
views
What is exactly sparse annotation?
What is exactly sparse annotation? Is it different from labeling images?
I've been reading a paper about vessel segmentation and have some issues understanding this part.
- 40.5k
51
votes
6
answers
3k
views
What is fuzzy logic?
I'm new to A.I. and I'd like to know in simple words, what is the fuzzy logic concept? How does it help, and when is it used?
- 332
1
vote
0
answers
42
views
References for Nvidia's DLSS
Nvidia's deep learning super-sampling is presented as an application of deep learning techniques to video-rendering in videogames.
Question: I'm asking for a technical reference that explains what is ...
- 133
1
vote
1
answer
2k
views
Using "softmax" (non-linear) vs "linear" activation function in Deep Reinforcement Learning
I am following the tutorial in this video: https://youtu.be/cO5g5qLrLSo which implements deep reinforcement learning (DQN) to balance cart pole in OpenAI default environment.
The DQN model looks like ...
- 32.1k
4
votes
0
answers
675
views
Is there any way and any reason why one would introduce a sparsity constraint on a deep auto-encoder?
Is there any way and any reason why one would introduce a sparsity constraint on a deep autoencoder?
In particular, in deep autoencoders, the first layer often has more units than the dimensionality ...
- 40.5k
5
votes
2
answers
663
views
What do the neural network's weights represent conceptually?
I understand how neural networks work and have studied their theory well.
My question is: On the whole, is there a clear understanding of how mutation occurs within a neural network from the input ...
- 40.5k
0
votes
0
answers
59
views
If the model always underfits, do I really need a larger model?
I train my neural network on random points generated for a data set that theoretically consists of approximately $1.8 * 10^{39}$ elements. I sample (generate) tens of thousands of random points on ...
- 53
3
votes
0
answers
278
views
Are there neural networks with (hard) constraints on the weights?
I don't know too much about Deep Learning, so my question might be silly. However, I was wondering whether there are NN architectures with some hard constraints on the weights of some layers. For ...
- 40.5k
9
votes
4
answers
12k
views
What could an oscillating training loss curve represent?
I tried to create a simple model that receives an $80 \times 130$ pixel image. I only had 35 images and 10 test images. I trained this model for a binary classification task. The architecture of the ...
- 1,283
4
votes
1
answer
781
views
What is the state-of-the-art algorithm for neural style transfer?
I've read the paper A Neural Algorithm of Artistic Style by Gatys et. al. and I find the application of neural style transfer very fun.
I also read that Exploring the structure of a real-time, ...
CommunityBot
- 1
2
votes
4
answers
160
views
Is there a way of pre-determining whether a CNN model will perform better than another?
I developed a CNN for image analysis. I've around 100K labeled images. I'm getting a accuracy around 85% and a validation accuracy around 82%, so it looks like the model generalize better than ...
- 903
1
vote
1
answer
435
views
Are there any animation tools available to visualise and simulate deep neural networks? [closed]
Deep learning researchers have to work with a lot of models. The models may include different types of Layers: They include convolutional neural network layers, recurrent neural network layers, batch ...
- 6,227
0
votes
1
answer
537
views
Dissection of a depth map
I am curious about how depth maps work. While searching I came across this website which contains some images and their depth maps. I took this depth map and tried to study it using a python pillow.
<...
- 350
2
votes
0
answers
619
views
How does back propagation adjust the hidden layers' weights and biases?
I'm new to neural networks and trying to figure out its fundamentals but I cannot fully understand the back propagation algorithm.
In back propagation, I understand we want to go backwards from the ...
0
votes
0
answers
530
views
Computational complexity of a CNN network
In the following network, the convolution operations of convolutional blocks are performed by three 1-D kernels with the sizes 8, 5, and 3 respectively along with stride equal to 1. The final network ...
- 1
1
vote
1
answer
99
views
Would this count as a Transfer Learning approach?
I have two datasets, Dataset 1(D1) and Dataset 2(D2). D1 has around 22000 samples, and D2 has around 8000 samples. What I am doing is that I train a Deep Neural Network model with around three layers ...
- 32.1k
3
votes
1
answer
546
views
What is meant by "stable training" of a deep learning model?
I have read it said that the "stable training" of a deep learning model is important. What is meant by "stable training" of a deep learning model?
- 40.5k
0
votes
1
answer
464
views
Number of classes vs number of parameters/layers?
How to estimate the number of parameters in CNN for object detection?
I know that there are some well-known architectures that was trained on a lot of data (AlexNet, ResNet, VGG, GoogleLeNet). But ...
- 552
2
votes
1
answer
331
views
Do we ever need more then 1 hidden layer in a binary classification problem with ANNs? If yes why?
I have read about the universal approximation theorem. So, why do we need more than 1 layer? Is it somehow computationally efficient to add layers instead of more neurons in the hidden layer?
- 903
11
votes
1
answer
4k
views
When should you not use the bias in a layer?
I'm not really that experienced with deep learning, and I've been looking at research code (mostly PyTorch) for deep neural networks, specifically GANs, and, in many cases, I see the authors setting <...
- 2,524
2
votes
1
answer
261
views
What is the difference between multi-head and normal output?
Let's say that I have a neural network with 2 heads. The first consists of X neurons. The second consists of Y neurons. I have these 2 heads because I want to predict 2 different variables. And I can ...
- 226
3
votes
1
answer
74
views
Are there deep neural networks that have inputs connected with deeper hidden layers?
Are there any architectures of deep neural networks that connect input neurons not only with the first hidden layer but also with deeper ones (red lines on the picture)?
If so could you give some ...
- 350
1
vote
1
answer
89
views
Should binary feature be in one or two columns in deep neural networks?
Let's assume I have a simple feedforward neural network whose input contains binary 0/1 features and output is also binary two classes.
Is it better, worse, or maybe totally indifferent, for every ...
- 350
1
vote
1
answer
265
views
Unable to 'learn' a rotational angle by parametrising the angle as a neural network layer
I'm trying to implement a neural network that can capture the drift in a measured angle as a way of dynamic calibration. i.e, I have a reference system that may change throughout the course of the ...
- 13
1
vote
0
answers
36
views
The MLP output of a neural network can be written as $\|x\|\|w_l\|\cos(\theta_l)$: why is the norm easier to maximize?
The MLP output of a neural network is a dot product between the weights and the input and therefore can be written as $\|x\|\|w_l\|\cos(\theta_l)$ (see this for more details), where $x$ is the input, $...
- 1,010