Questions tagged [deep-neural-networks]

For questions related to deep neural networks, which are artificial neural networks with "many" layers, where "many" can vary depending on the context.

Filter by
Sorted by
Tagged with
0 votes
0 answers
38 views

The end-to-end Training Process for Knowledge Distillation

I'm a bit confused on the complete training process for Knowledge Distillation. I was reading the Geoffrey Hinton "Distilling the Knowledge in a Neural Network" 2015 paper and some random ...
Chuu's user avatar
  • 1
0 votes
1 answer
54 views

What is the potential issue of nested neural networks

everyone. I am working on a nested neural network architecture. For the sake of better understanding my question, simply assume the loss is $L = G(k’) - H(k'')$ where $G$ and $H$ are two functions we ...
Zuba Tupaki's user avatar
1 vote
0 answers
32 views

Replicating conv autoencoder for anomaly detection, very blurry reconstructions

I’m trying to train an autoencoder on the hazelnut dataset of MVTec AD for reconstruction to detect anomalies. I’m am trying to replicate the results of this study: https://arxiv.org/pdf/2008.12977....
JeanMi's user avatar
  • 165
1 vote
0 answers
91 views

In the figure of Stable Diffusion, when does the switch part change?

In the illustration of Stable Diffusion, there is a concatenation part through Cross-Attention. Why is there a switch in the concatenation part?
diffusion stable's user avatar
1 vote
0 answers
73 views

Pointers to (deep) latent variable models that admit analytical approximations

I am aware that there is a plethora of deep generative models out there (e.g. variational autoencoders (VAE), GANs) that can model high-dimensional data as the images of latent variables under a non-...
ngiann's user avatar
  • 111
0 votes
0 answers
27 views

How to differentiate fuzzy matching from artificial intelligence [duplicate]

I am curious about the term "fuzzy matching," and whether it falls under the category of artificial intelligence. Specifically, when can we say that a website or system is using AI, and when ...
CollarKaniz's user avatar
1 vote
2 answers
96 views

How to do image classification with optional metadata?

I have a vanilla image classification problem. The image may optionally have some numerical metadata associated with it. We don't assume uniform availability of this metadata, i.e., the model should ...
Vardaan Pahuja's user avatar
0 votes
0 answers
28 views

Independent parameter update in backpropagation

When we calculate the gradient wrt to each paramters, we consider the other parameters remain constant, but the moment their is a change in any of the other parameters, shouldn't all the other changes ...
In progress...'s user avatar
1 vote
2 answers
222 views

Why is dot-product and not Euclidean distance used for attention?

In models using attention (eg Transformer architectures) we used scaled dot-product to measure similarity rather than (negative or inverse) Euclidean distance. Why is this the case? Does Layer ...
Betterthan Kwora's user avatar
0 votes
1 answer
43 views

Temporally Non-Aware RNN

I am trying to classify whether or not a specific object is in panoramic photos. The issue is, a panoramic photo can be any width, so the input to my neural network can't be fixed in that dimension. I'...
user avatar
0 votes
0 answers
19 views

Can we use bounding box cropping to avoid shortcut learning (achieve explainable AI)?

Deep neural networks sometime use shortcut features (pseudo correlation) to predict. For example, in cat-dog classification, the network may use the background information (e.g. floor, grass) as a ...
Mingzhou Liu's user avatar
0 votes
0 answers
21 views

Is it possible to combine SGD with an unsupervised learning approach effectively

Before I undertake quite a large project I would like to clarify whether my idea for training a multi-layer neural network will work. I plan to make an AI that can land a rocket from randomly ...
Gamaray's user avatar
0 votes
0 answers
15 views

Linear Discriminant Analysis on a transformed space

Let $S$ be a finite subset of a $\mathbb{R}^k$ partitioned into $N$ subsets $S_1, \ldots, S_N$ and let $n_j = |S_j|$. The between-groups sum of squares of the partition is defined as $$bSS(S_1,\ldots, ...
Alberto's user avatar
  • 101
1 vote
1 answer
42 views

How to handle large dimensionality differences between state and action inputs in a reinforcement learning predictor?

I'm currently writing code for a reward predictor function r=f(s,a) in reinforcement learning, where 's' is the state with 256 dimensions (the embedding dimension after visual input is processed by an ...
XiaoBanni's user avatar
0 votes
0 answers
65 views

How to tell an agent that some actions in the action space are currently not available in gym and the design of action space

I want to make a task allocation decision by reinforcement learning. Suppose there are N tasks to be allocated and M severs to serve these task. However, there is a constraint that one task should be ...
Reese's user avatar
  • 1
0 votes
1 answer
40 views

When training a DNN model, how can I make some training data points more important than others?

In other words, is it possible to assign "weights" to data points during model training? Is there a standard technique for it? It seems like it the math would be straightforward enough for ...
PlinyTheElder's user avatar
1 vote
0 answers
35 views

Can a Fully Connected Neural Network represent all Neural Networks of smaller size?

A fully connected Neural Network architecture can be characterized by a vector $\mathbf a = (a_0,a_1,\ldots,a_L)\in\mathbb N^{L+1}$ and an activation function $\sigma :\mathbb R\to\mathbb R$. In this ...
Stratos supports the strike's user avatar
0 votes
0 answers
27 views

How to identify location from a given input in a semantic way

I have a sample use case, where user will give us a xls file with some location data. Where may be location information is given in different ways , sometime its IATA code, sometimes its standard ...
Sujoy's user avatar
  • 1
2 votes
0 answers
21 views

In DQN how does the Q network not converge to the incorrect target? [duplicate]

Whenever you are doing reinforcement learning you periodically update the target network based on the weights of the Q network. While I do understand this helps create a stable target I do not ...
Stef's user avatar
  • 95
2 votes
0 answers
62 views

Is orthogonal initialization still useful when hidden layer sizes vary?

Pytorch's orthogonal initialization cites "Exact solutions to the nonlinear dynamics of learning in deep linear neural networks ", Saxe, A. et al. (2013), which gives as reason for the ...
Gabi's user avatar
  • 121
0 votes
1 answer
43 views

Classify sequence of flags

I am not able to find an answer to how I should classify a varying number of sequence of binary flags + other features. My data looks like this (these are events, so the order is important and I may ...
Zaba's user avatar
  • 1
0 votes
1 answer
75 views

Difference in gradient calculation for the last layer activation in neural networks

I'm currently working on implementing a neural network using the sigmoid activation function and the binary cross-entropy cost function. In my implementation, I've noticed that the gradient ...
gratus richard's user avatar
0 votes
1 answer
17 views

What architecture is used for deep quadruplet network for person re-identification

I am trying to implement the paper Beyond triplet loss: a deep quadruplet network for person re-identification. In the paper, they provide a figure (attached below) containing the network architecture,...
Varghese Kuruvilla's user avatar
0 votes
0 answers
24 views

Longer DNN training times when using evolutionary algorithms

I am comparing my deep neural network (DNN) performance when using 2 types of optimizers: gradient-based Adam (properly tuned) and a population-based optimization algorithm (e.g., genetic algorithm (...
knowledge_seeker's user avatar
2 votes
1 answer
87 views

Can models like chatGPT learn functions with infinite domain or range

Lets assume two types of prompt: A fixed prompt for which reasonable responses can be infinite. For example: > output a random number > output a palindrome ...
Sayam Qazi's user avatar
0 votes
0 answers
13 views

Do deep ensembles and regular ensembles coincide for classification tasks?

The deep ensemble paper https://arxiv.org/pdf/1612.01474.pdf introduces proper scoring rules for ensembles of NNs. Turns out that the likelihood is always a proper scoring rule. For regression tasks, ...
astrolollo's user avatar
3 votes
0 answers
22 views

Why does training converges when the norm of gradient increases?

This is from deep learning book by Ian Goodfellow and Yoshua Bengio and Aaron Courville. When training converges well, I thought the gradient should be at local minima. But the book says it often does ...
tesio's user avatar
  • 205
0 votes
1 answer
113 views

How vision models based on CNNs learn the relative positions of each pixel in the image?

A CNN model is based on a series of filters applied to an image. However, these filters can only "see" a small portion of the image and they have no information of the relative position of ...
IgnacioGaBo's user avatar
1 vote
1 answer
137 views

What should I do, reinforcement learning agent gives different result on every train?

I'm using PPO+LSTM to create a trading bot. The agent is trained on 3 years of data and tested on 1 year. Every time I train the agent with same set of hyper-parameters, I get very different results ...
ad124j2's user avatar
  • 11
0 votes
1 answer
44 views

How to improve classification accuracy in TF deep neural network model?

I need help in increasing the accuracy of a classification model using Neural Networks on Tensorflow. I am trying to train a model on sequential data ...
Fr_nkenstien's user avatar
1 vote
0 answers
119 views

What does Deep Q-Learning (DQL) do?

Hello :) I'm required to write a document where I describe what DQL does in short. This is what I wrote: DQL: instead of a Q-table, a DNN is used to approximate the Q-values for each action based on a ...
Ness's user avatar
  • 206
1 vote
1 answer
122 views

Formally, what are the layers in an Artificial Neural Network?

You may not believe it, but I am an ANN expert. Perhaps, for that reason, I am unable to grasp completely what the layers are in a Deep Forward Artificial Neural Network (DFANN). According to the Deep ...
neoglez's user avatar
  • 11
0 votes
2 answers
280 views

How to handle the size difference of highway network or residual network in cnn?

For highway network, it looks like this: For residual network, it looks like this: Pictures are from What is the name of this neural network architecture with layers that are also connected to non-...
liaoming999's user avatar
0 votes
0 answers
38 views

Does the number of epochs measure a correlation?

i have built a two-layers neural network (1000 => 1000) to predict a dynamical system driven by two real-world parameters. When using the first parameter as input to the first layer, training the ...
Bouarfa Mahi's user avatar
1 vote
2 answers
88 views

Does it make sense to build a two-layer neural network with a triangular weight matrix?

I need to implement a rule and have defined a lower triangular boolean mask for the weights that I want to keep static for a zero value. In which condition triangular weight matrix will be used?
Bouarfa Mahi's user avatar
1 vote
2 answers
1k views

Why should one expect the backward pass to take twice as long as the forward pass?

I have seen it stated that, as a rule of thumb, a backward pass in a neural network should take about twice as long as the forward pass. Examples: From DeepSpeed's Flops Profiler docs, the profiler: ...
user26866's user avatar
  • 111
0 votes
1 answer
523 views

Is it redunant to add more layers to a neural network with same number of neurons as the previous layer?

Lets say I have a neural network with three layers and the last layer has 3 outputs. If I added additional layer of 3 neurons to the end of the network, would that be a more powerful neural network? ...
Steven Sagona's user avatar
1 vote
1 answer
463 views

Why MSE and MAE yield poor results when used with gradient-based optimization for classification?

Deep learning book chapter 6: In 6.2.1.2 last paragraph: Unfortunately, mean squared error and mean absolute error often lead to poor results when used with gradient-based optimization. Some output ...
vivian.ai's user avatar
0 votes
3 answers
584 views

why validation accuracy be greater than training accuracy for deep learning models? [closed]

I hope you are well. I had a problem and didn't understand the answers given on questions similar to my question. If possible, please answer this problem in a simpler way. Val_acc : %99.4 _ Train_acc :...
maserati urm's user avatar
1 vote
1 answer
152 views

MobileNetV2 - Some particularities

So I was studying MobileNetV2 architecture and came across this table from the original paper that represents its architecture: Table Description: "Table 2: MobileNetV2 : Each line describes a ...
Blue Ross's user avatar
0 votes
1 answer
108 views

Training a neural network in full batch training

It is a trend in deep learning to train models using multi-batches, i.e., to show the model a subset of the entire dataset for each weight update. In some cases, as in continual learning, we see that ...
Alfred's user avatar
  • 165
1 vote
4 answers
157 views

Why, in deep learning, do we get computational power by going deeper?

I know by the expressiveness of a neural networks that it can be seen as a chain of function compositions, i.e. $g(f(.. z(x)..))$ and also that, if we go deep, we can approximate complex functions $f: ...
f_s's user avatar
  • 11
1 vote
0 answers
37 views

Why "Good Model" that performs great on holdout validation data fails on production data

I have this binary regression model that has ~500 futures with an unbalanced dataset with the following results. ...
Newbie's user avatar
  • 23
1 vote
1 answer
215 views

How to handle out-of-bound values in Production data?

So I have this model but the data may vary. And it is virtually impossible to always have the values in bounds. If I do I`d have to use larger period leading to concept shift which is worse. The ...
Newbie's user avatar
  • 23
1 vote
0 answers
86 views

How can I get an integer as output for continuous action space PPO reinforcement learning?

I have a huge discrete action space, the learning stability is not good. I'd like to move to continuous action space but the only output for my task can be a positive integer (let's say in the range 0 ...
NAnn's user avatar
  • 111
1 vote
1 answer
43 views

How to use information on a function to design a neural network learning that function?

I have a function $g$ that takes a vector $x$ of size $n$ and an integer $k$ in $1, \ldots, n$. I know this function is of the form $$g(x,k) = G\left(\sum_{i=1}^k f(x_{i})\right),$$ where $f$ and $G$ ...
Aristodog's user avatar
  • 111
1 vote
1 answer
39 views

How to identify important features in data?

I have a couple opportunities to write a paper, or papers over some of the neural networks I have made. I was wondering if there are anyways to figure out why the neural network classifies the data I ...
samuel fipps's user avatar
1 vote
0 answers
26 views

How Can We Create Neural Networks with Different Depths and Widths But Same Number of Parameters?

Right now I am doing a research project investigating how the depth of a Neural Network affects its capacity to learn. In order to do this, I wanted to test different Networks with the same number of ...
Vincent 's user avatar
0 votes
1 answer
250 views

How to compare memory requirements for tabular Q-learning vs deep neural network?

I want to compare the space complexity/memory requirement of tabular Q-learning v.s. deep neural Q-network (DQN). I think DQN would be faster and Q-table has a disadvantage at large table sizes but ...
knowledge_seeker's user avatar
1 vote
1 answer
43 views

How to preserve Markov Property in Deep Reinforcement Learning when using "mixup" or "mixreg"?

I've read through these two papers: (original about "mixup") https://arxiv.org/pdf/1710.09412.pdf (variant for RL, "mixreg") https://arxiv.org/pdf/2010.10814.pdf They are about a ...
Vladimir Belik's user avatar