Questions tagged [deep-neural-networks]

For questions related to deep neural networks, which are artificial neural networks with "many" layers, where "many" can vary depending on the context.

70 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
8 votes
1 answer
4k views

Does it make sense to use batch normalization in deep (stacked) or sparse auto-encoders?

Does it make sense to use batch normalization in deep (stacked) or sparse auto-encoders? I cannot find any resources for that. Is it safe to assume that, since it works for other DNNs, it will also ...
Glrs's user avatar
  • 231
6 votes
0 answers
178 views

Why Pixel RNN (Row LSTM) can capture triangular contexts?

I'm reading the paper Pixel Recurrent Neural Network. I have a question about Row LSTM. Why Row LSTM can capture triangular contexts? In this paper, the kernel of the one-dimensional convolution ...
musako's user avatar
  • 181
4 votes
2 answers
269 views

What kind of output should be used for predicting angles in DNNs?

I am building a model which predicts angles as output. What are the different kinds of outputs that can be used to predict angles? For example, output the angle in radians cyclic nature of the ...
Yashas's user avatar
  • 189
4 votes
0 answers
675 views

Is there any way and any reason why one would introduce a sparsity constraint on a deep auto-encoder?

Is there any way and any reason why one would introduce a sparsity constraint on a deep autoencoder? In particular, in deep autoencoders, the first layer often has more units than the dimensionality ...
MattSt's user avatar
  • 597
3 votes
0 answers
22 views

Why does training converges when the norm of gradient increases?

This is from deep learning book by Ian Goodfellow and Yoshua Bengio and Aaron Courville. When training converges well, I thought the gradient should be at local minima. But the book says it often does ...
tesio's user avatar
  • 205
3 votes
0 answers
271 views

Are there neural networks with (hard) constraints on the weights?

I don't know too much about Deep Learning, so my question might be silly. However, I was wondering whether there are NN architectures with some hard constraints on the weights of some layers. For ...
Onil90's user avatar
  • 183
3 votes
0 answers
74 views

What's up with Neural Stochastic Differential Equations from a practical standpoint?

I've spent a few days reading some of the new papers about Neural SDEs. For example, here is one from Tzen and Raginsky and here is one that came out simultaneously by Peluchetti and Favaro. There are ...
jeffery_the_wind's user avatar
3 votes
2 answers
353 views

How to build my own dataset and model for an LSTM neural network

I have a sort of mathematical problem and I'm not sure which model I should choose to make an LSTM neural network. Currently in my country, there is a system in which certain groups of researchers ...
LP0's user avatar
  • 83
2 votes
0 answers
62 views

Is orthogonal initialization still useful when hidden layer sizes vary?

Pytorch's orthogonal initialization cites "Exact solutions to the nonlinear dynamics of learning in deep linear neural networks ", Saxe, A. et al. (2013), which gives as reason for the ...
Gabi's user avatar
  • 121
2 votes
0 answers
993 views

How many layers do GPT-3, AlphaFold 2, and DALL-E 2 have?

Unsuccessfully, I tried to find out the "depth" (definition below) in large neural networks such as GPT-3, AlphaFold 2, and DALL-E 2. Formally, my question is about their computational graph:...
keyboardAnt's user avatar
2 votes
0 answers
592 views

How does back propagation adjust the hidden layers' weights and biases?

I'm new to neural networks and trying to figure out its fundamentals but I cannot fully understand the back propagation algorithm. In back propagation, I understand we want to go backwards from the ...
KiarashRzg's user avatar
2 votes
0 answers
61 views

Estimating dimensions to reduce input image size to in CNNs

Considering input images to a CNN that have a large dimension (e.g. 256X256), what are some possible methods to estimate the exact dimensions (e.g. 16X16 or 32X32) to which it can be condensed in the ...
Prishita Ray's user avatar
2 votes
0 answers
140 views

Are there any new weight initialization techniques for DNN published after 2015?

Considering weights initialization in my personal projects, I always used some standard techniques such as: Glorot (also known as Xavier) initialization (2010). Mertens initialization (2010). He ...
Stefano Barone's user avatar
2 votes
0 answers
129 views

Are there any commonly used discontinuous activation functions?

Are there any commonly used activation functions (e.g. that take values in $(0,.5)\cup (.5,1)$)? Preferably for classification? Why? I was looking for commonly used activation functions on Google, ...
ABIM's user avatar
  • 545
2 votes
0 answers
534 views

Relationship between model complexity (depth) and dataset size

I'm new to deep learning. I was wondering what's the relationship between a deep model complexity (e.g. total number of parameters, or depth) and the dataset size? Assuming I want to do a binary ...
Tina J's user avatar
  • 973
2 votes
0 answers
55 views

Which deep neural networks are appropriate for the detection of bombs?

This is a follow-up question from my previous post here about explosion detection. I gathered a dataset of explosions. As I'm new to Deep Learning in Keras, I'm trying to see what architecture best ...
Tina J's user avatar
  • 973
2 votes
1 answer
163 views

Spikes in of Train and Test error

I learn a DNN for image recognition. During each epoch, I calculate mean loss in the training set. After each epoch, I calculate loss and number of errors over both training and test set. The problem ...
user31264's user avatar
  • 203
2 votes
0 answers
73 views

How to define cost function for custom nonlinear functions?

For logistic regression, the Cost function is defined as: \begin{equation} Cost(h_{\theta}(x)-y) = -ylog(h_{\theta}(x))-(1-y)log(1-h_{\theta}(x)) \end{equation} I now have a nonlinear function \begin{...
yfliuuu's user avatar
  • 51
2 votes
0 answers
83 views

How would I implement this New Type of NN

CIO NN CIO NN stands for Controller Input Output Nerual Network note due to a typo the "nearon" means "neron" For this we have to redefine the Nearon 2 Inputs 2 Outputs 4 Weights (each input and ...
X3R0's user avatar
  • 211
1 vote
0 answers
32 views

Replicating conv autoencoder for anomaly detection, very blurry reconstructions

I’m trying to train an autoencoder on the hazelnut dataset of MVTec AD for reconstruction to detect anomalies. I’m am trying to replicate the results of this study: https://arxiv.org/pdf/2008.12977....
JeanMi's user avatar
  • 165
1 vote
0 answers
91 views

In the figure of Stable Diffusion, when does the switch part change?

In the illustration of Stable Diffusion, there is a concatenation part through Cross-Attention. Why is there a switch in the concatenation part?
diffusion stable's user avatar
1 vote
0 answers
73 views

Pointers to (deep) latent variable models that admit analytical approximations

I am aware that there is a plethora of deep generative models out there (e.g. variational autoencoders (VAE), GANs) that can model high-dimensional data as the images of latent variables under a non-...
ngiann's user avatar
  • 111
1 vote
0 answers
35 views

Can a Fully Connected Neural Network represent all Neural Networks of smaller size?

A fully connected Neural Network architecture can be characterized by a vector $\mathbf a = (a_0,a_1,\ldots,a_L)\in\mathbb N^{L+1}$ and an activation function $\sigma :\mathbb R\to\mathbb R$. In this ...
Stratos supports the strike's user avatar
1 vote
1 answer
137 views

What should I do, reinforcement learning agent gives different result on every train?

I'm using PPO+LSTM to create a trading bot. The agent is trained on 3 years of data and tested on 1 year. Every time I train the agent with same set of hyper-parameters, I get very different results ...
ad124j2's user avatar
  • 11
1 vote
0 answers
119 views

What does Deep Q-Learning (DQL) do?

Hello :) I'm required to write a document where I describe what DQL does in short. This is what I wrote: DQL: instead of a Q-table, a DNN is used to approximate the Q-values for each action based on a ...
Ness's user avatar
  • 206
1 vote
2 answers
88 views

Does it make sense to build a two-layer neural network with a triangular weight matrix?

I need to implement a rule and have defined a lower triangular boolean mask for the weights that I want to keep static for a zero value. In which condition triangular weight matrix will be used?
Bouarfa Mahi's user avatar
1 vote
0 answers
37 views

Why "Good Model" that performs great on holdout validation data fails on production data

I have this binary regression model that has ~500 futures with an unbalanced dataset with the following results. ...
Newbie's user avatar
  • 23
1 vote
0 answers
86 views

How can I get an integer as output for continuous action space PPO reinforcement learning?

I have a huge discrete action space, the learning stability is not good. I'd like to move to continuous action space but the only output for my task can be a positive integer (let's say in the range 0 ...
NAnn's user avatar
  • 111
1 vote
1 answer
43 views

How to use information on a function to design a neural network learning that function?

I have a function $g$ that takes a vector $x$ of size $n$ and an integer $k$ in $1, \ldots, n$. I know this function is of the form $$g(x,k) = G\left(\sum_{i=1}^k f(x_{i})\right),$$ where $f$ and $G$ ...
Aristodog's user avatar
  • 111
1 vote
0 answers
26 views

How Can We Create Neural Networks with Different Depths and Widths But Same Number of Parameters?

Right now I am doing a research project investigating how the depth of a Neural Network affects its capacity to learn. In order to do this, I wanted to test different Networks with the same number of ...
Vincent 's user avatar
1 vote
0 answers
41 views

References for Nvidia's DLSS

Nvidia's deep learning super-sampling is presented as an application of deep learning techniques to video-rendering in videogames. Question: I'm asking for a technical reference that explains what is ...
Ramiro Hum-Sah's user avatar
1 vote
1 answer
66 views

A neural network to learn the connection between two totally different type of images

I have a dataset of two different type of images. Say, I have images of a person and his all 10 fingerprints. I want to create a relation between them to predict one from another. How I can do that ...
Prapon's user avatar
  • 11
1 vote
0 answers
36 views

The MLP output of a neural network can be written as $\|x\|\|w_l\|\cos(\theta_l)$: why is the norm easier to maximize?

The MLP output of a neural network is a dot product between the weights and the input and therefore can be written as $\|x\|\|w_l\|\cos(\theta_l)$ (see this for more details), where $x$ is the input, $...
user8714896's user avatar
1 vote
0 answers
46 views

SAGAN - is there a mistake in the original paper?

in the original paper the following scheme of the self-attention appears: https://arxiv.org/pdf/1805.08318.pdf In a later overview: https://arxiv.org/pdf/1906.01529.pdf this scheme appears: ...
Ilya.K.'s user avatar
  • 111
1 vote
0 answers
346 views

DQN Agent with a 2D matrix as input in Keras

I have a Reinforcement Learning environment where the state is a 2D matrix with 0s and 1s (only one column with the value of 1 in each row). Example: ...
Ness's user avatar
  • 206
1 vote
0 answers
41 views

How to understand this NN architecture?

I was reading a paper Multi-Agent Reinforcement Learning for Adaptive User Association in Dynamic mmWave Networks and I was stuck understanding the deep neural network architecture that was used. The ...
zdm's user avatar
  • 301
1 vote
0 answers
43 views

How can one be sure that a particular neural network architecture would work?

Traditionally, when working with tabular data, one can be sure(or at least know) that a model works because the included features could explain a target variable, say "Price of a ticket" ...
Naveen Reddy Marthala's user avatar
1 vote
0 answers
52 views

Advantages of training Neural Networks based on analytic success criteria

What is the reason to train a Neural Network to estimate a task's success (i.e. robotic grasp planning) using a simulator that is based on analytic grasp quality metrics? Isn't a perfectly trained NN ...
EmVee's user avatar
  • 11
1 vote
0 answers
143 views

Understanding the role of the target network in this DQN algorithm

I've found online this interesting algorithm: From what I understand reading this algorithm, I can't figure out why I should "perform the opposite action" and consequently storing that second ...
unter_983's user avatar
  • 331
1 vote
0 answers
28 views

How can I find the similar non-zero connections between different levels of sparsity of the same network?

I am pruning a neural network (CNN and Dense) and for different sparsity levels, I have different sub-networks. Say for sparsity levels of 20%, 40%, 60% and 80%, I have 4 different sub-networks. Now, ...
Arun's user avatar
  • 225
1 vote
0 answers
41 views

Steps to train and re-train a good model

I'm still a bit new to deep learning. What I'm still struggling, is what is the best practice in re-training a good model over time? I've trained a deep model for my binary classification problem (...
Tina J's user avatar
  • 973
1 vote
0 answers
39 views

Post-classification after inference

I designed a fire detection using Deep Learning based classification approach. In my training dataset, I have both fire and fire smokes are supposed to be detected (all under "fire"; mostly real fires ...
Tina J's user avatar
  • 973
1 vote
0 answers
106 views

Should I use deep learning to solve my task?

I need to predict the performance (CPI cycles-per-instruction) of 90 machines for the next hour (or day). Each machine has a thousand records (e.g. CPU and memory usage). Currently, I am using a ...
jou's user avatar
  • 63
1 vote
0 answers
18 views

Camera pose to environment Mapping

I would like to teach a model the environment of a room. I'm doing so by mapping a camera pose (x, y, z, q0, q1, q2, q3) to its corresponding image; where x, y, z represent location in Cartesian ...
Ijlal's user avatar
  • 11
1 vote
0 answers
41 views

One end to end Neural network or many task-specific ones?

Is it better to train one neural network for a dispersed labeled data with large number of classes or first classify data by unsupervised learning then train each part by a separate NN? I mean by ...
Sajad.sni's user avatar
1 vote
0 answers
45 views

What is the feasible neural network structure that can learn to identify types of trajectory of moving dots?

I have multiple image sequences, each of which contains an animation of two moving dots. The trajectory of the dots in a sequence is always cyclic (not necessarily circular). There are two types of ...
shg's user avatar
  • 11
1 vote
0 answers
56 views

How to handle varying length of inputs that represent dependencies and recursivity in deep neural networks in case of regression?

I wanna solve a problem of regression to predict a factor. I decide to go with Deep Neural Networks as solution for my problem. The features in this problem represent loop characteristic such us loop ...
DINA TAKLIT's user avatar
1 vote
2 answers
1k views

How to create Partially Connected NNs with prespecified connections using Tensorflow?

I'd like to implement a partially connected neural network with ~3 to 4 hidden layers (a sparse deep neural network?) where I can specify which node connects to which node from the previous/next layer....
Pınar Demetci's user avatar
1 vote
0 answers
35 views

Using two generative adversarial nets to classify articles - what is a good approach?

I'm trying to create a deep learning network to classify news article based on the text and associated image. The idea comes from a novel use of GANs to classify based on generated data. My approach ...
elksie5000's user avatar
1 vote
0 answers
91 views

Confidence interval around a DNN prediction

I am facing a problem and do not know whether it is even solvable: I want to predict the behaviour of a system using a DNN, say a CNN, in the sense that I want to predict the time and intensity of a ...
Leon's user avatar
  • 173