Questions tagged [neural-networks]

For questions about a artificial networks, such as MLPs, CNNs, RNNs, LSTM, and GRU networks, their variants or any other AI system components that qualify as a neural networks in that they are, in part, inspired by biological neural networks.

Filter by
Sorted by
Tagged with
3
votes
1answer
107 views

What is numerical stability?

I came across the phrase "numerical stability" several times. But almost in the same context. I encountered this word mostly in the analytical formula for batch normalization. $$y = \dfrac{...
0
votes
0answers
8 views

Why do we transform feature vectors in attention modules for CNNs

If we have a set of feature maps with dimensions [B, C, H, W] (batch, channel, height, width), why do we transform our feature maps before we calculate their affinity/correlation in attention ...
1
vote
1answer
441 views

Does the input layer have bias and are there bias neurons?

I have seen two different representations of neural networks when it comes to bias. Consider a "simple" neural network, with just an input layer, a hidden layer and an output layer. To ...
2
votes
2answers
54 views

Does the output layer in a deep neural network need an activation function?

I have enrolled in a course that uses only one hidden layer, and that is the only layer that has activation functions. The model can be visualized as follows: and here is a PyTorch implementation: <...
4
votes
3answers
141 views

Why neural networks tend to be trained to recognize multiple things instead of just one?

I was watching this series: https://www.youtube.com/watch?v=aircAruvnKk The series demonstrates neural networks by building a simple number recognizing network. It got me thinking: Why neural networks ...
1
vote
2answers
202 views

Why the cost/loss starts to increase for some iterations during the training phase?

I am trying to build a recurrent neural network from scratch. It's a very simple model. I am trying to train it to predict two words (dogs and gods). While training, the value of cost function starts ...
0
votes
0answers
41 views

When does a batch normalization layer becomes active?

Let us assume your dataset has $n$ training samples each of size $s$ and you divided them into $k$ batches for training. Then each batch has $n_k = \dfrac{n}{k}$ training samples. Batch normalization ...
0
votes
1answer
181 views

What is the best neural network architecture for this problem?

I built a three-layer neural network (first is 1D convolutional and the remaining two are linear). It takes an input of 5 angles in radians, and outputs two numbers from 0 to 1, which are respectively ...
0
votes
0answers
29 views

How to measure the significance of an input feature for the output of a linear layer in a neural network

Suppose I have a simple linear layer $y = xA^T + b$ that is part of a neural network trained on some dataset. The weight matrix $A$ for this layer has the shape ...
0
votes
0answers
44 views

What are the numbers thar are useful (may need to be stored) other than parameters of a model?

Consider the following method related to buffers in PyTorch ...
6
votes
2answers
451 views

Neural network for data visualization

At my work, we're currently doing some research into data visualisation for highly interconnected data, basically graphs. We've been implementing all sorts of different layouts and trying to see which ...
4
votes
1answer
51 views

In the field of Deep Learning research, what considerations do researchers take into account when inventing new neural network models?

I am not a researcher, but I am curious to know what considerations are relevant to take into account during research for the invention of a new neural network model, and what relevant knowledge ...
1
vote
2answers
35 views

Is "Pruning" only applicable to CNNs?

What Is Neural Network Pruning And Why Is It Important Today? The above article only talks about Convolutional Neural Networks: One of the first methods of pruning is pruning entire convolutional ...
1
vote
1answer
260 views

Why is my validation/test accuracy higher than my training accuracy

Is this due to my dropout layers being disabled during evaluation? I'm classifying the CIFAR-10 dataset with a CNN using the Keras library. There are 50000 samples in the training set; I'm using a ...
2
votes
2answers
58 views

Why not make the training set and validation set one if their roles are similar?

If the validation set is used to tune the hyperparameters and the training set adjusts the weights, why don't they be one thing as they have a similar role, as in improving the model?
2
votes
0answers
20 views

ReLU function converging to local optimum in one case and diverging in the other one

I implemented a simple neural network with 1 hidden layer. I used ReLU as activation function for the hidden layer and the output layer just uses the linear function. To check my implementation I ...
0
votes
0answers
12 views

Deep NN architecture for predicting a matrix from a matrix and list of floats

I am trying to predict a matrix (size RxC) based on an input matrix (size RxC) and a list of floats ...
1
vote
1answer
20 views

Is reconciling shape discrepancies the only purpose of padding?

Padding is a technique used in some of the domains of artificial intelligence. Data is generally available in different shapes. But in order to pass the data as input to a model in deep learning, the ...
2
votes
1answer
72 views

How to create a neural network from a set of equations?

Say I have these equations: $$x_1 = x_2 + 2y_1 + b$$ $$x_2 = y_2 + c$$ $$y_1 = z + a$$ $$y_2 = y_3 + d$$ $$z = z_1 + e$$ $x_1$ depends on $x_2$ (depends on $y_2$ (depends on $y_3$)) and $y_1$ (depends ...
1
vote
0answers
19 views

Is there any existing mechanism that allows us to pass input from randomly selected layers of neural network per iteration?

Consider the following neural network with $\ell$ layers. $$i_0 \rightarrow h_1 \rightarrow h_2 \rightarrow h_3 \cdots \rightarrow h_{\ell-1} \rightarrow o_{\ell} ,$$ where $i, h, o$ stands for ...
3
votes
1answer
127 views

Is batch normalization not suitable for non-gaussian input?

I generate some non-Gaussian data, and use two kinds of DNN models, one with BN and the other without BN. I find that the model DNN with BN can't predict well. The codes is shown as follow: <...
1
vote
0answers
41 views

Why doesn't anyone use reinforcement learning to find the best possible alternative to backpropagation?

To be clear, I'm very uninformed on the topic of alternative learning algorithms to backprop, all my knowledge comes from articles like these: lets-not-stop-at-backprop backprop-alternatives we-need-a-...
2
votes
0answers
25 views

Methodologies for passing the best samples for a neural network to learn

Just an idea I am sure I read in a book some time ago, but I can't remember the name. Given a very large dataset and a neural network (or anything that can learn via something like stochastic gradient ...
2
votes
0answers
38 views

How much research, approximately, is done in ANNs?

Does someone know where can I find information about how much research, nowadays, is done in ANNs? I've checked in this document Redes Neuronales: Conceptos básicos y aplicaciones, Universidad ...
2
votes
1answer
113 views

Price difference predictions curve almost vanished

With a team, we are studying how it is possible to predict the price movement with high-frequency. Instead of predicting the price directly, we have decided to try predicting price difference as well ...
0
votes
1answer
42 views

Where does batch normalization layers present in a neural network?

Batch normalization is a procedure widely used to train neural networks. Mean and standard deviation are calculated in this step of training. Since we train neural network by dividing training data ...
0
votes
0answers
35 views

What are the input and output gradients in PyTorch?

Suppose I want to train a neural network with $m-$length inputs of form $x = [x_1, x_2, x_3, \cdots, x_m]$ and $n-$length outputs of form $y = [y_1, y_2, y_3, \cdots, y_n]$. Let the number of ...
1
vote
2answers
105 views

How to extract parameters from a text using AI/NLP

lets say I have three texts: "make a heading that says hello word" "make a heading of hello world" "create heading consist of hello world" How can I fetch those groups ...
1
vote
1answer
87 views

How does the regression layer in the localization network of a spatial transformer work?

I am trying to understand the spatial transformer network mentioned in this paper https://papers.nips.cc/paper/5854-spatial-transformer-networks.pdf. I am clear about the last two stages of the ...
5
votes
1answer
145 views

How do big companies, like Facebook, model individuals and their interaction?

As a layman in AI, I want to get an idea of how big data players, like Facebook, model individuals (of which they have so many data). There are two scenarios I can imagine: Neural networks build ...
2
votes
1answer
185 views

Anomaly Detection in distributed system using generated log file

I am developing an AI tool for anomaly detection in a distributed system.  The system supports an interface that combines several individual logs into a single log file generating approx. 7000 entries/...
1
vote
1answer
100 views

How to properly optimize shared network between actor and critic?

I'm building an actor-critic reinforcment learning algorithm to solve environments. I want to use a single encoder to find representation of my environment. When I share the encoder with the actor ...
3
votes
2answers
96 views

Is there any difference between affine transformation and linear transformation?

Consider the following statements from A Simple Custom Module of PyTorch's documentation To get started, let’s look at a simpler, custom version of PyTorch’s Linear module. This module applies an ...
0
votes
0answers
9 views

How to approach receipt interpretation with neural networks

I have never worked with neural networks before, but I want to learn and I have a specific use case that I would want to solve (or at least try to). I have developed software for multiple years, in a ...
1
vote
1answer
91 views

Why does my "entropy generation" RNN do so badly?

I'm new to relatively RNNs, and I'm trying to train generative and guessing neural networks to produce sequences of real numbers that look random. My architecture looks like this (each "circle&...
2
votes
1answer
232 views

Is the dropout technique specific only to neural networks?

In one Udemy course was mentioned that "dropout is unique to neural networks". However, I remember an example of decision trees where nodes that are not participating in the overall result ...
0
votes
1answer
77 views

classification of unseen classes of image in open set classification

I have a scanned image, and they need to be classified in one of the pre-defined image classes, so that it can be sorted. However, the problem is the open nature of the classes. At testing time, new ...
1
vote
3answers
1k views

How to create Partially Connected NNs with prespecified connections using Tensorflow?

I'd like to implement a partially connected neural network with ~3 to 4 hidden layers (a sparse deep neural network?) where I can specify which node connects to which node from the previous/next layer....
2
votes
1answer
98 views

Can you explain me this CNN architecture?

I am starting to get my head around convolutional neural networks, and I have been working with the CIFAR-10 dataset and some research papers that used it. In one of these papers, they mention a ...
1
vote
1answer
40 views

How is the convolution operation connected to neural networks?

I've been reading up on the convolution operation and neural networks. I understand that the convolution operation is defined as: $$(f * g)(t)=\int_{-\infty}^{\infty} f(\tau) g(t-\tau) d \tau$$ The ...
0
votes
1answer
10 views

Can we modelize an RNN by an ANN that takes precedent output as a part of input?

Is it possible to consider an RNN as a classical feedforward neural network that just take the precedent output as a part of the input ?
0
votes
0answers
12 views

Comparing results of different image splicing methods on a part of the CASIA 2.0 dataset

So I am working on an image splicing detection algorithm using ResNet-50 model. I am using the CASIA 2.0 dataset which consists of 7491 Authentic images and 5123 Fake images. However out of the fake ...
1
vote
2answers
2k views

Understanding the loss function in deep Q-learning

I am trying to understand how deep Q learning (DQN) works. To my current understanding, each $Q(s, a)$ functions is estimated to be a function of a feature vector of its state $\phi$(s) and the weight ...
0
votes
1answer
38 views

What are the labels of EMNIST?

I'm using pytorch and downloading the EMNIST dataset using ...
1
vote
0answers
31 views

Do the terms multi-task and multi-output refer to the same thing in the context of deep learning?

Do the terms multi-task and multi-output refer to the same thing in the context of deep learning (with neural networks)? For example, do neural networks for multi-task learning use multiple outputs? ...
4
votes
2answers
2k views

What is the general procedure to use and train neural networks for multi-class classification?

I am very new to machine learning. I am following the course offered by Andrew Ng. I am very confused about how we train our neural network for multi-class classification. Let's say we have $K$ ...
0
votes
0answers
41 views

RLLib - What exactly do the avail_action and action_embed_size represent? How do they work with the action_mask to phase out invalid actions?

So, I'm fairly new to reinforcement learning and I needed some help/explanations as to what the action_mask and avail_action fields alongside the action_embed_size actually mean in RLlib (the ...
2
votes
0answers
38 views

Identifying rotating and resizing letters with background noise

I'm trying to complete a captcha, and here is what it looks like: Between captchas the calligraphy of the letters is the same, but the letters may be resized and rotated. And the background noise (...
0
votes
0answers
22 views

Does Keras or Tensorflow calculate backpropagation?

I am using tensorflow 2.1.0. Does Keras or Tensorflow calculate backpropagation? Where can I find the code for backpropagation in Keras and Tensorflow? Thanks, Jianqiao Huang

1 2
3
4 5
44