Questions tagged [deep-neural-networks]

For questions related to deep neural networks, which are artificial neural networks with "many" layers, where "many" can vary depending on the context.

59 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
6 votes
0 answers
4k views

Does it make sense to use batch normalization in deep (stacked) or sparse auto-encoders?

Does it make sense to use batch normalization in deep (stacked) or sparse auto-encoders? I cannot find any resources for that. Is it safe to assume that, since it works for other DNNs, it will also ...
user avatar
  • 211
5 votes
0 answers
111 views

Why Pixel RNN (Row LSTM) can capture triangular contexts?

I'm reading the paper Pixel Recurrent Neural Network. I have a question about Row LSTM. Why Row LSTM can capture triangular contexts? In this paper, the kernel of the one-dimensional convolution ...
user avatar
4 votes
2 answers
85 views

What kind of output should be used for predicting angles in DNNs?

I am building a model which predicts angles as output. What are the different kinds of outputs that can be used to predict angles? For example, output the angle in radians cyclic nature of the ...
user avatar
  • 179
4 votes
0 answers
653 views

Is there any way and any reason why one would introduce a sparsity constraint on a deep auto-encoder?

Is there any way and any reason why one would introduce a sparsity constraint on a deep autoencoder? In particular, in deep autoencoders, the first layer often has more units than the dimensionality ...
user avatar
  • 557
3 votes
0 answers
66 views

Are there neural networks with (hard) constraints on the weights?

I don't know too much about Deep Learning, so my question might be silly. However, I was wondering whether there are NN architectures with some hard constraints on the weights of some layers. For ...
user avatar
  • 163
3 votes
0 answers
51 views

What's up with Neural Stochastic Differential Equations from a practical standpoint?

I've spent a few days reading some of the new papers about Neural SDEs. For example, here is one from Tzen and Raginsky and here is one that came out simultaneously by Peluchetti and Favaro. There are ...
user avatar
3 votes
1 answer
175 views

Is batch normalization not suitable for non-gaussian input?

I generate some non-Gaussian data, and use two kinds of DNN models, one with BN and the other without BN. I find that the model DNN with BN can't predict well. The codes is shown as follow: <...
user avatar
  • 131
3 votes
2 answers
274 views

How to build my own dataset and model for an LSTM neural network

I have a sort of mathematical problem and I'm not sure which model I should choose to make an LSTM neural network. Currently in my country, there is a system in which certain groups of researchers ...
user avatar
  • 83
2 votes
0 answers
74 views

How does back propagation adjust the hidden layers' weights and biases?

I'm new to neural networks and trying to figure out its fundamentals but I cannot fully understand the back propagation algorithm. In back propagation, I understand we want to go backwards from the ...
user avatar
2 votes
0 answers
51 views

Estimating dimensions to reduce input image size to in CNNs

Considering input images to a CNN that have a large dimension (e.g. 256X256), what are some possible methods to estimate the exact dimensions (e.g. 16X16 or 32X32) to which it can be condensed in the ...
user avatar
2 votes
0 answers
95 views

Are there any new weight initialization techniques for DNN published after 2015?

Considering weights initialization in my personal projects, I always used some standard techniques such as: Glorot (also known as Xavier) initialization (2010). Mertens initialization (2010). He ...
user avatar
2 votes
0 answers
80 views

Are there any commonly used discontinuous activation functions?

Are there any commonly used activation functions (e.g. that take values in $(0,.5)\cup (.5,1)$)? Preferably for classification? Why? I was looking for commonly used activation functions on Google, ...
user avatar
  • 495
2 votes
0 answers
52 views

Which deep neural networks are appropriate for the detection of bombs?

This is a follow-up question from my previous post here about explosion detection. I gathered a dataset of explosions. As I'm new to Deep Learning in Keras, I'm trying to see what architecture best ...
user avatar
  • 963
2 votes
1 answer
108 views

Spikes in of Train and Test error

I learn a DNN for image recognition. During each epoch, I calculate mean loss in the training set. After each epoch, I calculate loss and number of errors over both training and test set. The problem ...
user avatar
  • 203
2 votes
0 answers
70 views

How to define cost function for custom nonlinear functions?

For logistic regression, the Cost function is defined as: \begin{equation} Cost(h_{\theta}(x)-y) = -ylog(h_{\theta}(x))-(1-y)log(1-h_{\theta}(x)) \end{equation} I now have a nonlinear function \begin{...
user avatar
  • 51
2 votes
0 answers
81 views

How would I implement this New Type of NN

CIO NN CIO NN stands for Controller Input Output Nerual Network note due to a typo the "nearon" means "neron" For this we have to redefine the Nearon 2 Inputs 2 Outputs 4 Weights (each input and ...
user avatar
1 vote
0 answers
17 views

How Can We Create Neural Networks with Different Depths and Widths But Same Number of Parameters?

Right now I am doing a research project investigating how the depth of a Neural Network affects its capacity to learn. In order to do this, I wanted to test different Networks with the same number of ...
user avatar
1 vote
0 answers
74 views

How many layers do GPT-3, AlphaFold 2, and DALL-E 2 have?

Unsuccessfully, I tried to find out the "depth" (definition below) in large neural networks such as GPT-3, AlphaFold 2, and DALL-E 2. Formally, my question is about their computational graph:...
user avatar
1 vote
0 answers
22 views

References for Nvidia's DLSS

Nvidia's deep learning super-sampling is presented as an application of deep learning techniques to video-rendering in videogames. Question: I'm asking for a technical reference that explains what is ...
user avatar
1 vote
0 answers
16 views

Output representation for a neural network to learn grid-based game with multiple units

I have a round based game played on a grid map with multiple units that I would like to control in some fashion using neural network (NN). All of the units are moved at once. Each unit can move in any ...
user avatar
  • 159
1 vote
1 answer
134 views

Multi label classification on non binary labels with pytorch

I am working on a project consisting of medical images and a huge dataset of multi-label and non-binary labels/outcomes ( sex, blood pressure, age and 40 more ). Would be the best approach to hard ...
user avatar
1 vote
0 answers
28 views

How Tesla and other companies use outputs from neural networks to drive the car?

Here is the short description of Tesla Autopilot AI: https://www.tesla.com/autopilotAI And here are some videos about how Tesla uses neural networks: Andrej Karpathy - AI for Full-Self Driving at ...
user avatar
  • 183
1 vote
0 answers
26 views

The MLP output of a neural network can be written as $\|x\|\|w_l\|\cos(\theta_l)$: why is the norm easier to maximize?

The MLP output of a neural network is a dot product between the weights and the input and therefore can be written as $\|x\|\|w_l\|\cos(\theta_l)$ (see this for more details), where $x$ is the input, $...
user avatar
1 vote
0 answers
40 views

SAGAN - is there a mistake in the original paper?

in the original paper the following scheme of the self-attention appears: https://arxiv.org/pdf/1805.08318.pdf In a later overview: https://arxiv.org/pdf/1906.01529.pdf this scheme appears: ...
user avatar
  • 111
1 vote
0 answers
209 views

DQN Agent with a 2D matrix as input in Keras

I have a Reinforcement Learning environment where the state is a 2D matrix with 0s and 1s (only one column with the value of 1 in each row). Example: ...
user avatar
  • 147
1 vote
0 answers
37 views

How to understand this NN architecture?

I was reading a paper Multi-Agent Reinforcement Learning for Adaptive User Association in Dynamic mmWave Networks and I was stuck understanding the deep neural network architecture that was used. The ...
user avatar
  • 247
1 vote
0 answers
42 views

How can one be sure that a particular neural network architecture would work?

Traditionally, when working with tabular data, one can be sure(or at least know) that a model works because the included features could explain a target variable, say "Price of a ticket" ...
user avatar
1 vote
0 answers
47 views

Advantages of training Neural Networks based on analytic success criteria

What is the reason to train a Neural Network to estimate a task's success (i.e. robotic grasp planning) using a simulator that is based on analytic grasp quality metrics? Isn't a perfectly trained NN ...
user avatar
  • 11
1 vote
1 answer
288 views

Can we use a pre trained Encoder (BERT, XLM ) with a Decoder (GPT, Transformer-XL) to build a Chatbot instead of Language Translation?

I was wondering if the BART or T5 models can do the task of generating sentences in English. Most of the models I have mentioned ...
user avatar
  • 233
1 vote
0 answers
105 views

Understanding the role of the target network in this DQN algorithm

I've found online this interesting algorithm: From what I understand reading this algorithm, I can't figure out why I should "perform the opposite action" and consequently storing that second ...
user avatar
  • 281
1 vote
0 answers
27 views

How can I find the similar non-zero connections between different levels of sparsity of the same network?

I am pruning a neural network (CNN and Dense) and for different sparsity levels, I have different sub-networks. Say for sparsity levels of 20%, 40%, 60% and 80%, I have 4 different sub-networks. Now, ...
user avatar
  • 185
1 vote
0 answers
32 views

Steps to train and re-train a good model

I'm still a bit new to deep learning. What I'm still struggling, is what is the best practice in re-training a good model over time? I've trained a deep model for my binary classification problem (...
user avatar
  • 963
1 vote
0 answers
28 views

Post-classification after inference

I designed a fire detection using Deep Learning based classification approach. In my training dataset, I have both fire and fire smokes are supposed to be detected (all under "fire"; mostly real fires ...
user avatar
  • 963
1 vote
0 answers
144 views

Relationship between model complexity (depth) and dataset size

I'm new to deep learning. I was wondering what's the relationship between a deep model complexity (e.g. total number of parameters, or depth) and the dataset size? Assuming I want to do a binary ...
user avatar
  • 963
1 vote
0 answers
106 views

Should I use deep learning to solve my task?

I need to predict the performance (CPI cycles-per-instruction) of 90 machines for the next hour (or day). Each machine has a thousand records (e.g. CPU and memory usage). Currently, I am using a ...
user avatar
  • 63
1 vote
0 answers
17 views

Camera pose to environment Mapping

I would like to teach a model the environment of a room. I'm doing so by mapping a camera pose (x, y, z, q0, q1, q2, q3) to its corresponding image; where x, y, z represent location in Cartesian ...
user avatar
  • 11
1 vote
0 answers
39 views

One end to end Neural network or many task-specific ones?

Is it better to train one neural network for a dispersed labeled data with large number of classes or first classify data by unsupervised learning then train each part by a separate NN? I mean by ...
user avatar
1 vote
0 answers
44 views

What is the feasible neural network structure that can learn to identify types of trajectory of moving dots?

I have multiple image sequences, each of which contains an animation of two moving dots. The trajectory of the dots in a sequence is always cyclic (not necessarily circular). There are two types of ...
user avatar
  • 11
1 vote
0 answers
49 views

How to handle varying length of inputs that represent dependencies and recursivity in deep neural networks in case of regression?

I wanna solve a problem of regression to predict a factor. I decide to go with Deep Neural Networks as solution for my problem. The features in this problem represent loop characteristic such us loop ...
user avatar
1 vote
3 answers
1k views

How to create Partially Connected NNs with prespecified connections using Tensorflow?

I'd like to implement a partially connected neural network with ~3 to 4 hidden layers (a sparse deep neural network?) where I can specify which node connects to which node from the previous/next layer....
user avatar
1 vote
0 answers
34 views

Using two generative adversarial nets to classify articles - what is a good approach?

I'm trying to create a deep learning network to classify news article based on the text and associated image. The idea comes from a novel use of GANs to classify based on generated data. My approach ...
user avatar
1 vote
0 answers
87 views

Confidence interval around a DNN prediction

I am facing a problem and do not know whether it is even solvable: I want to predict the behaviour of a system using a DNN, say a CNN, in the sense that I want to predict the time and intensity of a ...
user avatar
  • 163
1 vote
0 answers
27 views

Are there a finite set of computable functions constructing deep neural network which can form or implement any c.e. function or computable function?

Are there a finite set of computable functions constructing deep neural network which can form or implement any c.e. function or computable function? Or does there exist a finite set of computable ...
user avatar
0 votes
0 answers
4 views

Is it possible to embed a neural network layer into decision tree/random forest?

I want to do a classification task. I designed a customed layer for it. I also want to try decision tree/random forest, but as far as I know there is no way to embed my layer into a decsion tree/...
user avatar
0 votes
0 answers
55 views

What is the time complexity of Deep VGG-16 Net?

I would like to compute the time complexity (in Big O notation) of Deep Vgg16 Net by feeding forward 1000 images with size 224x224x3 and retrieving features from the second fully connected layer 'fc7' ...
user avatar
0 votes
0 answers
31 views

DNN architecture for regression with large output layers

I'm facing a supervised regression problem where I need to predict the outcome of the numerical simulation of a physical process. Each simulation-sample outputs a 256x256x6 tensor of real numbers ...
user avatar
0 votes
2 answers
144 views

How to count overlapping objects with neural networks

Consider the following task to be solved by a neural network: Given a $N\times N$ pixel grid with up to $M$ objects drawn on it, either squares (9 pixels) or diamonds (5 pixels): square    diamond The ...
user avatar
0 votes
0 answers
33 views

Differential Privacy measures of any trained ML model

I wonder if there is any practical way to assess/measure the level of privacy that a trained ML model has using Pytorch? I know that there are different techniques which helps to guarantee certain ...
user avatar
0 votes
0 answers
50 views

If the model always underfits, do I really need a larger model?

I train my neural network on random points generated for a data set that theoretically consists of approximately $1.8 * 10^{39}$ elements. I sample (generate) tens of thousands of random points on ...
user avatar
0 votes
0 answers
14 views

What to infer If a parameter value is the same as I initialized while training?

I am running a neural network model. There are several parameters. I initialized a new parameter to a value (say k). When I observe its change during the training, it is not changing like other ...
user avatar
  • 3,099