Questions tagged [deep-neural-networks]
For questions related to deep neural networks, which are artificial neural networks with "many" layers, where "many" can vary depending on the context.
197
questions
0
votes
1
answer
43
views
What is the potential issue of nested neural networks
everyone. I am working on a nested neural network architecture. For the sake of better understanding my question, simply assume the loss is
$L = G(k’) - H(k'')$
where $G$ and $H$ are two functions we ...
1
vote
0
answers
27
views
Replicating conv autoencoder for anomaly detection, very blurry reconstructions
I’m trying to train an autoencoder on the hazelnut dataset of MVTec AD for reconstruction to detect anomalies.
I’m am trying to replicate the results of this study:
https://arxiv.org/pdf/2008.12977....
- 165
1
vote
0
answers
55
views
In the figure of Stable Diffusion, when does the switch part change?
In the illustration of Stable Diffusion, there is a concatenation part through Cross-Attention. Why is there a switch in the concatenation part?
1
vote
0
answers
57
views
Pointers to (deep) latent variable models that admit analytical approximations
I am aware that there is a plethora of deep generative models out there (e.g. variational autoencoders (VAE), GANs) that can model high-dimensional data as the images of latent variables under a non-...
- 111
0
votes
0
answers
27
views
How to differentiate fuzzy matching from artificial intelligence [duplicate]
I am curious about the term "fuzzy matching," and whether it falls under the category of artificial intelligence. Specifically, when can we say that a website or system is using AI, and when ...
1
vote
2
answers
42
views
How to do image classification with optional metadata?
I have a vanilla image classification problem. The image may optionally have some numerical metadata associated with it. We don't assume uniform availability of this metadata, i.e., the model should ...
0
votes
0
answers
20
views
Independent parameter update in backpropagation
When we calculate the gradient wrt to each paramters, we consider the other parameters remain constant, but the moment their is a change in any of the other parameters, shouldn't all the other changes ...
1
vote
2
answers
113
views
Why is dot-product and not Euclidean distance used for attention?
In models using attention (eg Transformer architectures) we used scaled dot-product to measure similarity rather than (negative or inverse) Euclidean distance. Why is this the case?
Does Layer ...
- 121
0
votes
1
answer
42
views
Temporally Non-Aware RNN
I am trying to classify whether or not a specific object is in panoramic photos. The issue is, a panoramic photo can be any width, so the input to my neural network can't be fixed in that dimension.
I'...
user55176
0
votes
0
answers
19
views
Can we use bounding box cropping to avoid shortcut learning (achieve explainable AI)?
Deep neural networks sometime use shortcut features (pseudo correlation) to predict.
For example, in cat-dog classification, the network may use the background information (e.g. floor, grass) as a ...
- 111
0
votes
0
answers
18
views
Is it possible to combine SGD with an unsupervised learning approach effectively
Before I undertake quite a large project I would like to clarify whether my idea for training a multi-layer neural network will work. I plan to make an AI that can land a rocket from randomly ...
- 1
0
votes
0
answers
13
views
Linear Discriminant Analysis on a transformed space
Let $S$ be a finite subset of a $\mathbb{R}^k$ partitioned into $N$ subsets $S_1, \ldots, S_N$ and let $n_j = |S_j|$. The between-groups sum of squares of the partition is defined as
$$bSS(S_1,\ldots, ...
- 101
1
vote
1
answer
36
views
How to handle large dimensionality differences between state and action inputs in a reinforcement learning predictor?
I'm currently writing code for a reward predictor function r=f(s,a) in reinforcement learning, where 's' is the state with 256 dimensions (the embedding dimension after visual input is processed by an ...
- 51
0
votes
0
answers
44
views
How to tell an agent that some actions in the action space are currently not available in gym and the design of action space
I want to make a task allocation decision by reinforcement learning. Suppose there are N tasks to be allocated and M severs to serve these task.
However, there is a constraint that one task should be ...
- 1
0
votes
1
answer
38
views
When training a DNN model, how can I make some training data points more important than others?
In other words, is it possible to assign "weights" to data points during model training? Is there a standard technique for it?
It seems like it the math would be straightforward enough for ...
- 103
1
vote
0
answers
32
views
Can a Fully Connected Neural Network represent all Neural Networks of smaller size?
A fully connected Neural Network architecture can be characterized by a vector $\mathbf a = (a_0,a_1,\ldots,a_L)\in\mathbb N^{L+1}$ and an activation function $\sigma :\mathbb R\to\mathbb R$. In this ...
0
votes
0
answers
26
views
How to identify location from a given input in a semantic way
I have a sample use case, where user will give us a xls file with some location data. Where may be location information is given in different ways , sometime its IATA code, sometimes its standard ...
- 1
2
votes
0
answers
21
views
In DQN how does the Q network not converge to the incorrect target? [duplicate]
Whenever you are doing reinforcement learning you periodically update the target network based on the weights of the Q network. While I do understand this helps create a stable target I do not ...
- 53
2
votes
0
answers
40
views
Is orthogonal initialization still useful when hidden layer sizes vary?
Pytorch's orthogonal initialization cites "Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
", Saxe, A. et al. (2013), which gives as reason for the ...
- 121
0
votes
1
answer
38
views
Classify sequence of flags
I am not able to find an answer to how I should classify a varying number of sequence of binary flags + other features. My data looks like this (these are events, so the order is important and I may ...
- 1
0
votes
0
answers
34
views
Difference in gradient calculation for the last layer activation in neural networks
I'm currently working on implementing a neural network using the sigmoid activation function and the binary cross-entropy cost function. In my implementation, I've noticed that the gradient ...
0
votes
1
answer
17
views
What architecture is used for deep quadruplet network for person re-identification
I am trying to implement the paper Beyond triplet loss: a deep quadruplet network for person re-identification. In the paper, they provide a figure (attached below) containing the network architecture,...
0
votes
0
answers
24
views
Longer DNN training times when using evolutionary algorithms
I am comparing my deep neural network (DNN) performance when using 2 types of optimizers: gradient-based Adam (properly tuned) and a population-based optimization algorithm (e.g., genetic algorithm (...
2
votes
1
answer
84
views
Can models like chatGPT learn functions with infinite domain or range
Lets assume two types of prompt:
A fixed prompt for which reasonable responses can be infinite. For example:
> output a random number
> output a palindrome
...
- 23
0
votes
0
answers
13
views
Do deep ensembles and regular ensembles coincide for classification tasks?
The deep ensemble paper https://arxiv.org/pdf/1612.01474.pdf introduces proper scoring rules for ensembles of NNs. Turns out that the likelihood is always a proper scoring rule.
For regression tasks, ...
3
votes
0
answers
20
views
Why does training converges when the norm of gradient increases?
This is from deep learning book by Ian Goodfellow and Yoshua Bengio and Aaron Courville.
When training converges well, I thought the gradient should be at local minima. But the book says it often does ...
- 195
0
votes
1
answer
87
views
How vision models based on CNNs learn the relative positions of each pixel in the image?
A CNN model is based on a series of filters applied to an image. However, these filters can only "see" a small portion of the image and they have no information of the relative position of ...
- 63
1
vote
1
answer
121
views
What should I do, reinforcement learning agent gives different result on every train?
I'm using PPO+LSTM to create a trading bot. The agent is trained on 3 years of data and tested on 1 year. Every time I train the agent with same set of hyper-parameters, I get very different results ...
- 11
0
votes
1
answer
38
views
How to improve classification accuracy in TF deep neural network model?
I need help in increasing the accuracy of a classification model using Neural Networks on Tensorflow.
I am trying to train a model on sequential data ...
- 457
1
vote
0
answers
99
views
What does Deep Q-Learning (DQL) do?
Hello :) I'm required to write a document where I describe what DQL does in short.
This is what I wrote:
DQL: instead of a Q-table, a DNN is used to approximate the Q-values for each action based on a ...
- 206
1
vote
1
answer
118
views
Formally, what are the layers in an Artificial Neural Network?
You may not believe it, but I am an ANN expert. Perhaps, for that reason, I am unable to grasp completely what the layers are in a Deep Forward Artificial Neural Network (DFANN).
According to the Deep ...
- 11
0
votes
0
answers
32
views
Is it possible to create a distance estimation model from something like the KITTI dataset?
I am unsure about how to word this question correctly, edits appreciated.
I am trying to create a neural network model that can predict distance from camera feed. And I am doing it by feeding actual ...
0
votes
2
answers
191
views
How to handle the size difference of highway network or residual network in cnn?
For highway network, it looks like this:
For residual network, it looks like this:
Pictures are from What is the name of this neural network architecture with layers that are also connected to non-...
- 53
0
votes
0
answers
30
views
What are the inputs of a neural network when learning a difference equation?
The time series y[n] is the solution of the difference equation
...
- 11
0
votes
0
answers
38
views
Does the number of epochs measure a correlation?
i have built a two-layers neural network (1000 => 1000) to predict a dynamical system driven by two real-world parameters.
When using the first parameter as input to the first layer, training the ...
- 11
1
vote
2
answers
67
views
Does it make sense to build a two-layer neural network with a triangular weight matrix?
I need to implement a rule and have defined a lower triangular boolean mask for the weights that I want to keep static for a zero value. In which condition triangular weight matrix will be used?
- 11
0
votes
0
answers
23
views
What are the best ways to preprocess landmarks to train a Neural Network?
I'd like to know, generally speaking, which are the most useful ways to preprocess landmarks to use as training set for a Deep Neural Network. Since they're put in a 3D space, would that be enough ...
- 1
1
vote
2
answers
1k
views
Why should one expect the backward pass to take twice as long as the forward pass?
I have seen it stated that, as a rule of thumb, a backward pass in a neural network should take about twice as long as the forward pass. Examples:
From DeepSpeed's Flops Profiler docs, the profiler:
...
- 111
0
votes
1
answer
364
views
Is it redunant to add more layers to a neural network with same number of neurons as the previous layer?
Lets say I have a neural network with three layers and the last layer has 3 outputs.
If I added additional layer of 3 neurons to the end of the network, would that be a more powerful neural network?
...
- 221
1
vote
1
answer
400
views
Why MSE and MAE yield poor results when used with gradient-based optimization for classification?
Deep learning book chapter 6: In 6.2.1.2 last paragraph:
Unfortunately, mean squared error and mean absolute error often lead to poor results when used with gradient-based optimization. Some output ...
- 27
0
votes
3
answers
413
views
why validation accuracy be greater than training accuracy for deep learning models? [closed]
I hope you are well.
I had a problem and didn't understand the answers given on questions similar to my question.
If possible, please answer this problem in a simpler way.
Val_acc : %99.4 _
Train_acc :...
1
vote
1
answer
132
views
MobileNetV2 - Some particularities
So I was studying MobileNetV2 architecture and came across this table from the original paper that represents its architecture:
Table Description: "Table 2: MobileNetV2 : Each line describes a ...
- 27
0
votes
1
answer
90
views
Training a neural network in full batch training
It is a trend in deep learning to train models using multi-batches, i.e., to show the model a subset of the entire dataset for each weight update. In some cases, as in continual learning, we see that ...
- 165
1
vote
4
answers
155
views
Why, in deep learning, do we get computational power by going deeper?
I know by the expressiveness of a neural networks that it can be seen as a chain of function compositions, i.e. $g(f(.. z(x)..))$ and also that, if we go deep, we can approximate complex functions $f: ...
- 11
1
vote
0
answers
35
views
Why "Good Model" that performs great on holdout validation data fails on production data
I have this binary regression model that has ~500 futures with an unbalanced dataset with the following results.
...
- 23
1
vote
1
answer
182
views
How to handle out-of-bound values in Production data?
So I have this model but the data may vary. And it is virtually impossible to always have the values in bounds. If I do I`d have to use larger period leading to concept shift which is worse.
The ...
- 23
1
vote
0
answers
74
views
How can I get an integer as output for continuous action space PPO reinforcement learning?
I have a huge discrete action space, the learning stability is not good. I'd like to move to continuous action space but the only output for my task can be a positive integer (let's say in the range 0 ...
- 111
1
vote
1
answer
41
views
How to use information on a function to design a neural network learning that function?
I have a function $g$ that takes a vector $x$ of size $n$ and an integer $k$ in $1, \ldots, n$. I know this function is of the form
$$g(x,k) = G\left(\sum_{i=1}^k f(x_{i})\right),$$ where $f$ and $G$ ...
- 111
1
vote
1
answer
38
views
How to identify important features in data?
I have a couple opportunities to write a paper, or papers over some of the neural networks I have made.
I was wondering if there are anyways to figure out why the neural network classifies the data I ...
- 11
1
vote
0
answers
25
views
How Can We Create Neural Networks with Different Depths and Widths But Same Number of Parameters?
Right now I am doing a research project investigating how the depth of a Neural Network affects its capacity to learn. In order to do this, I wanted to test different Networks with the same number of ...
- 11