Questions tagged [deep-neural-networks]
For questions related to deep neural networks, which are artificial neural networks with "many" layers, where "many" can vary depending on the context.
201
questions
0
votes
0
answers
7
views
Do I need model calibration/feature normalization for reliable path integrated gradient/sampled Shapley feature attribution in a dnn model?
Are model calibration and feature normalization required for path integrated gradient and sampled Shapley-based feature attribution analyses to work properly in a deep neural network model?
I read ...
0
votes
0
answers
12
views
Why does the ECE increase after applying temperature scaling?
I am planning to apply temperature scaling to the output of my model. To determine the optimal temperature, I employed cross-validation and utilized the code available at this link: https://github.com/...
- 1
1
vote
0
answers
14
views
Compare two songs content using Audio Spectogram Transformer
I'm trying to establish a similarity metric between two songs. To do this I'm using the AST model on HuggingFace. This model basically works in a way very similar to a ViT but applied to spectograms ...
- 111
0
votes
0
answers
11
views
Optuna Hyperband Algorithm Not Following Expected Model Training Scheme
I have observed an issue while using the Hyperband algorithm in Optuna. According to the Hyperband algorithm, when min_resources = 5, max_resources = 20, and reduction_factor = 2, the search should ...
0
votes
0
answers
13
views
input_tensor is not connected to anything in onnx but can still do inference. How?
I converted an effdet model trained on custom dataset from tensorflow to onnx and tried to further convert it to tensorrt engine file. However, while doing so, I got uint8 issue (tensorrt does not ...
0
votes
1
answer
27
views
CNN multioutput regression architecture modification
I am working on a regression task where the model has to predict two values at the same time. The idea is that the dataset consists of 16 features, where the first 8 features represent the first value ...
0
votes
0
answers
18
views
Ideal method for finding function that satisfies a set of constraints
I'm new to playing around with deep learning. I'm trying to find a function that satisfies a bunch of constraints, and looking for tips on how to better my approach.
Let $F:X\times Y \mapsto [0,1]$, ...
- 1
0
votes
0
answers
14
views
Enhancing Soil Moisture Predictions Using Multimodal Data Integration in Agriculture
I am exploring an interdisciplinary research area involving multimodal data, focusing on agriculture. My study incorporates both visual and tabular data: crop and soil images from three distinct ...
- 1
0
votes
0
answers
40
views
The end-to-end Training Process for Knowledge Distillation
I'm a bit confused on the complete training process for Knowledge Distillation. I was reading the Geoffrey Hinton "Distilling the Knowledge in a Neural Network" 2015 paper and some random ...
- 1
0
votes
1
answer
59
views
What is the potential issue of nested neural networks
everyone. I am working on a nested neural network architecture. For the sake of better understanding my question, simply assume the loss is
$L = G(k’) - H(k'')$
where $G$ and $H$ are two functions we ...
1
vote
0
answers
34
views
Replicating conv autoencoder for anomaly detection, very blurry reconstructions
I’m trying to train an autoencoder on the hazelnut dataset of MVTec AD for reconstruction to detect anomalies.
I’m am trying to replicate the results of this study:
https://arxiv.org/pdf/2008.12977....
- 165
1
vote
0
answers
109
views
In the figure of Stable Diffusion, when does the switch part change?
In the illustration of Stable Diffusion, there is a concatenation part through Cross-Attention. Why is there a switch in the concatenation part?
1
vote
0
answers
75
views
Pointers to (deep) latent variable models that admit analytical approximations
I am aware that there is a plethora of deep generative models out there (e.g. variational autoencoders (VAE), GANs) that can model high-dimensional data as the images of latent variables under a non-...
- 111
0
votes
0
answers
27
views
How to differentiate fuzzy matching from artificial intelligence [duplicate]
I am curious about the term "fuzzy matching," and whether it falls under the category of artificial intelligence. Specifically, when can we say that a website or system is using AI, and when ...
1
vote
2
answers
132
views
How to do image classification with optional metadata?
I have a vanilla image classification problem. The image may optionally have some numerical metadata associated with it. We don't assume uniform availability of this metadata, i.e., the model should ...
0
votes
0
answers
30
views
Independent parameter update in backpropagation
When we calculate the gradient wrt to each paramters, we consider the other parameters remain constant, but the moment their is a change in any of the other parameters, shouldn't all the other changes ...
1
vote
2
answers
279
views
Why is dot-product and not Euclidean distance used for attention?
In models using attention (eg Transformer architectures) we used scaled dot-product to measure similarity rather than (negative or inverse) Euclidean distance. Why is this the case?
Does Layer ...
- 121
0
votes
1
answer
43
views
Temporally Non-Aware RNN
I am trying to classify whether or not a specific object is in panoramic photos. The issue is, a panoramic photo can be any width, so the input to my neural network can't be fixed in that dimension.
I'...
user55176
0
votes
0
answers
20
views
Can we use bounding box cropping to avoid shortcut learning (achieve explainable AI)?
Deep neural networks sometime use shortcut features (pseudo correlation) to predict.
For example, in cat-dog classification, the network may use the background information (e.g. floor, grass) as a ...
- 111
0
votes
0
answers
27
views
Is it possible to combine SGD with an unsupervised learning approach effectively
Before I undertake quite a large project I would like to clarify whether my idea for training a multi-layer neural network will work. I plan to make an AI that can land a rocket from randomly ...
- 1
0
votes
0
answers
15
views
Linear Discriminant Analysis on a transformed space
Let $S$ be a finite subset of a $\mathbb{R}^k$ partitioned into $N$ subsets $S_1, \ldots, S_N$ and let $n_j = |S_j|$. The between-groups sum of squares of the partition is defined as
$$bSS(S_1,\ldots, ...
- 101
1
vote
1
answer
42
views
How to handle large dimensionality differences between state and action inputs in a reinforcement learning predictor?
I'm currently writing code for a reward predictor function r=f(s,a) in reinforcement learning, where 's' is the state with 256 dimensions (the embedding dimension after visual input is processed by an ...
- 51
0
votes
0
answers
66
views
How to tell an agent that some actions in the action space are currently not available in gym and the design of action space
I want to make a task allocation decision by reinforcement learning. Suppose there are N tasks to be allocated and M severs to serve these task.
However, there is a constraint that one task should be ...
- 1
0
votes
1
answer
42
views
When training a DNN model, how can I make some training data points more important than others?
In other words, is it possible to assign "weights" to data points during model training? Is there a standard technique for it?
It seems like it the math would be straightforward enough for ...
- 103
1
vote
0
answers
38
views
Can a Fully Connected Neural Network represent all Neural Networks of smaller size?
A fully connected Neural Network architecture can be characterized by a vector $\mathbf a = (a_0,a_1,\ldots,a_L)\in\mathbb N^{L+1}$ and an activation function $\sigma :\mathbb R\to\mathbb R$. In this ...
0
votes
0
answers
28
views
How to identify location from a given input in a semantic way
I have a sample use case, where user will give us a xls file with some location data. Where may be location information is given in different ways , sometime its IATA code, sometimes its standard ...
- 1
2
votes
0
answers
21
views
In DQN how does the Q network not converge to the incorrect target? [duplicate]
Whenever you are doing reinforcement learning you periodically update the target network based on the weights of the Q network. While I do understand this helps create a stable target I do not ...
- 95
2
votes
0
answers
65
views
Is orthogonal initialization still useful when hidden layer sizes vary?
Pytorch's orthogonal initialization cites "Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
", Saxe, A. et al. (2013), which gives as reason for the ...
- 121
0
votes
1
answer
44
views
Classify sequence of flags
I am not able to find an answer to how I should classify a varying number of sequence of binary flags + other features. My data looks like this (these are events, so the order is important and I may ...
- 1
0
votes
1
answer
86
views
Difference in gradient calculation for the last layer activation in neural networks
I'm currently working on implementing a neural network using the sigmoid activation function and the binary cross-entropy cost function. In my implementation, I've noticed that the gradient ...
0
votes
1
answer
17
views
What architecture is used for deep quadruplet network for person re-identification
I am trying to implement the paper Beyond triplet loss: a deep quadruplet network for person re-identification. In the paper, they provide a figure (attached below) containing the network architecture,...
0
votes
0
answers
25
views
Longer DNN training times when using evolutionary algorithms
I am comparing my deep neural network (DNN) performance when using 2 types of optimizers: gradient-based Adam (properly tuned) and a population-based optimization algorithm (e.g., genetic algorithm (...
2
votes
1
answer
87
views
Can models like chatGPT learn functions with infinite domain or range
Lets assume two types of prompt:
A fixed prompt for which reasonable responses can be infinite. For example:
> output a random number
> output a palindrome
...
- 23
3
votes
0
answers
26
views
Why does training converges when the norm of gradient increases?
This is from deep learning book by Ian Goodfellow and Yoshua Bengio and Aaron Courville.
When training converges well, I thought the gradient should be at local minima. But the book says it often does ...
- 205
0
votes
1
answer
123
views
How vision models based on CNNs learn the relative positions of each pixel in the image?
A CNN model is based on a series of filters applied to an image. However, these filters can only "see" a small portion of the image and they have no information of the relative position of ...
- 63
1
vote
2
answers
177
views
What should I do, reinforcement learning agent gives different result on every train?
I'm using PPO+LSTM to create a trading bot. The agent is trained on 3 years of data and tested on 1 year. Every time I train the agent with same set of hyper-parameters, I get very different results ...
- 11
0
votes
1
answer
45
views
How to improve classification accuracy in TF deep neural network model?
I need help in increasing the accuracy of a classification model using Neural Networks on Tensorflow.
I am trying to train a model on sequential data ...
- 457
1
vote
0
answers
125
views
What does Deep Q-Learning (DQL) do?
Hello :) I'm required to write a document where I describe what DQL does in short.
This is what I wrote:
DQL: instead of a Q-table, a DNN is used to approximate the Q-values for each action based on a ...
- 206
1
vote
1
answer
125
views
Formally, what are the layers in an Artificial Neural Network?
You may not believe it, but I am an ANN expert. Perhaps, for that reason, I am unable to grasp completely what the layers are in a Deep Forward Artificial Neural Network (DFANN).
According to the Deep ...
- 11
0
votes
2
answers
368
views
How to handle the size difference of highway network or residual network in cnn?
For highway network, it looks like this:
For residual network, it looks like this:
Pictures are from What is the name of this neural network architecture with layers that are also connected to non-...
- 53
0
votes
0
answers
38
views
Does the number of epochs measure a correlation?
i have built a two-layers neural network (1000 => 1000) to predict a dynamical system driven by two real-world parameters.
When using the first parameter as input to the first layer, training the ...
- 11
1
vote
2
answers
102
views
Does it make sense to build a two-layer neural network with a triangular weight matrix?
I need to implement a rule and have defined a lower triangular boolean mask for the weights that I want to keep static for a zero value. In which condition triangular weight matrix will be used?
- 11
1
vote
2
answers
2k
views
Why should one expect the backward pass to take twice as long as the forward pass?
I have seen it stated that, as a rule of thumb, a backward pass in a neural network should take about twice as long as the forward pass. Examples:
From DeepSpeed's Flops Profiler docs, the profiler:
...
- 111
0
votes
1
answer
612
views
Is it redunant to add more layers to a neural network with same number of neurons as the previous layer?
Lets say I have a neural network with three layers and the last layer has 3 outputs.
If I added additional layer of 3 neurons to the end of the network, would that be a more powerful neural network?
...
- 221
1
vote
1
answer
502
views
Why MSE and MAE yield poor results when used with gradient-based optimization for classification?
Deep learning book chapter 6: In 6.2.1.2 last paragraph:
Unfortunately, mean squared error and mean absolute error often lead to poor results when used with gradient-based optimization. Some output ...
- 27
0
votes
3
answers
724
views
why validation accuracy be greater than training accuracy for deep learning models? [closed]
I hope you are well.
I had a problem and didn't understand the answers given on questions similar to my question.
If possible, please answer this problem in a simpler way.
Val_acc : %99.4 _
Train_acc :...
1
vote
1
answer
158
views
MobileNetV2 - Some particularities
So I was studying MobileNetV2 architecture and came across this table from the original paper that represents its architecture:
Table Description: "Table 2: MobileNetV2 : Each line describes a ...
- 37
0
votes
1
answer
113
views
Training a neural network in full batch training
It is a trend in deep learning to train models using multi-batches, i.e., to show the model a subset of the entire dataset for each weight update. In some cases, as in continual learning, we see that ...
- 165
1
vote
4
answers
159
views
Why, in deep learning, do we get computational power by going deeper?
I know by the expressiveness of a neural networks that it can be seen as a chain of function compositions, i.e. $g(f(.. z(x)..))$ and also that, if we go deep, we can approximate complex functions $f: ...
- 11
1
vote
0
answers
39
views
Why "Good Model" that performs great on holdout validation data fails on production data
I have this binary regression model that has ~500 futures with an unbalanced dataset with the following results.
...
- 23