Questions tagged [neural-networks]
For questions about artificial networks, such as MLPs, CNNs, RNNs, LSTM, and GRU networks, their variants or any other AI system components that qualify as a neural networks in that they are, in part, inspired by biological neural networks.
877
questions with no upvoted or accepted answers
10
votes
1
answer
935
views
Are Cellular Neural Networks one type of Neural Networks?
I am researching Cellular Neural Networks and have already read Chua's two articles (1988). In cellular neural networks, a cell is only in relation with its neighbors. So it is easy to use them for ...
- 283
8
votes
1
answer
188
views
What is the impact of using multiple BMUs for self-organizing maps?
Here's a sort of a conceptual question. I was implementing a SOM algorithm to better understand its variations and parameters. I got curious about one bit: the BMU (best matching unit == the neuron ...
- 181
7
votes
1
answer
155
views
How does the network know which objects to track in the paper "Label-Free Supervision of Neural Networks with Physics and Domain Knowledge"?
I was reading the paper Label-Free Supervision of Neural Networks with Physics and Domain Knowledge, published at AAAI 2017, which won the best paper award.
I understand the math and it makes sense. ...
- 171
6
votes
1
answer
113
views
It is possible to use deep learning to give approximate solutions to NP-hard graph theory problems?
It is possible to use deep learning to give approximate solutions to NP-hard graph theory problems?
If we take, for example, the travelling salesman problem (or the dominating set problem). Let's say ...
- 181
6
votes
2
answers
197
views
How to classify human actions?
I'm quite new to machine learning (I followed the Coursera course of Andrew Ng and now starting deeplearning.ai courses).
I want to classify human actions real-time like:
Left-arm bended
Arm above ...
- 161
6
votes
1
answer
235
views
How do big companies, like Facebook, model individuals and their interaction?
As a layman in AI, I want to get an idea of how big data players, like Facebook, model individuals (of which they have so many data).
There are two scenarios I can imagine:
Neural networks build ...
5
votes
1
answer
2k
views
Which other loss functions for hierarchical multi-label classification could I use?
I am looking to try different loss functions for a hierarchical multi-label classification problem. So far, I have been training different models or submodels like multilayer perceptron (MLP) branch ...
- 163
5
votes
4
answers
1k
views
Use Machine/Deep Learning to Guess a String
I want to be able to input a block of text and then have it guess a string within a predefined range (i.e. a string that starts with three letters and ends with five numbers like "XXX12345", etc). ...
- 59
4
votes
0
answers
200
views
how do you train a neural network to determine shortest path in a 4-node graph
Suppose I have the following graph:
...
- 141
4
votes
0
answers
68
views
Why does a neural network struggle to solve this simple problem?
Consider the following problem:
Given a vector x of size dim with values between 0 and 1 (exclusive), determine if ...
- 201
4
votes
0
answers
829
views
What is a neural network compatibility function?
Typically, a neural network parameterized by weights $\mathbf{W}$ is
a function from an input $x$ to an output $y$. The network has
an associated compatibility function $\Psi(y; x, \mathbf{W}) \...
- 199
4
votes
2
answers
247
views
Which neural network can I use to solve this constrained optimisation problem?
Let $\mathcal{S}$ be the training data set, where each input $u^i \in \mathcal{S}$ has $d$ features.
I want to design an ANN so that the cost function below is minimized (the sum of the square of ...
- 199
4
votes
0
answers
70
views
Can the quality of randomness in neural network initialization affect model fitting?
This is a topic I have been arguing about for some time now with my colleagues, maybe you could also voice your opinion about it.
Artificial neural networks use random weight initialization within a ...
- 41
4
votes
0
answers
90
views
Visualizing the Loss Landscape of Neural Nets: Meaning of the word 'filter'?
I found myself scratching my head when I read the following phrase in the paper Visualizing the Loss Landscape of Neural Nets:
To remove this scaling effect, we plot loss functions using filter-wise ...
- 41
4
votes
0
answers
176
views
Why does sigmoid saturation prevent signal flow through the neuron?
As per these slides on page 35:
Sigmoids saturate and kill gradients.
when the neuron's activation saturates at either tail of 0 or 1, the gradient at these regions is almost zero.
the gradient and ...
- 193
4
votes
0
answers
273
views
Why are most commonly used activation functions continuous?
I have come to notice that the most commonly used activation functions are continuous. Is there any specific reason behind this? Results such as this paper have worked on training networks with ...
- 564
4
votes
0
answers
83
views
When do two identical neural networks have uncorrelated errors?
In Chapter 9, section 9.1.6, Raul Rojas describes how committees of networks can reduce the prediction error by training N identical neural networks and averaging the results.
If $f_i$ are the ...
- 227
4
votes
0
answers
69
views
When is using weight regularization bad?
Regularization of weights (e.g. L1 or L2) keeps them small and standardized, which can help reduce data overfitting. From this article, regularization sounds favorable in many cases, but is it always ...
- 793
4
votes
0
answers
227
views
Has the logistic map ever been used as an activation function?
I find the logistic map absolutely fascinating. Both in itself (because I love fractal) and because it is observed in nature (see: https://www.youtube.com/watch?v=ovJcsL7vyrk).
I'm wondering if anyone ...
- 163
4
votes
1
answer
192
views
Is batch learning with gradient descent equivalent to "rehearsal" in incremental learning?
I am learning about incremental learning and read that rehearsal learning is retraining with old data. In essence, isn't this the exact same thing as batch learning (with stochastic gradient descent)? ...
- 213
4
votes
0
answers
2k
views
Should batch normalisation be applied before or after ReLU?
I know that there has been some discussion about this (e.g. here and here), but I can't seem to find consensus.
The crucial thing that I haven't seen mentioned in these discussions is that applying ...
- 171
4
votes
0
answers
291
views
Understanding the results of "Visualizing and Understanding Convolutional Networks"
I am trying to understand the results of the paper Visualizing and Understanding Convolutional Networks, in particular the following image:
What are these 3x3 blocks and their 9 cells representing?
...
- 91
4
votes
0
answers
39
views
How do weights changes handles during back-propagation when there are unknown labels
I have a question about how weights are updated during back-propagation for some of my samples that have unknown labels (please note, unknown, not missing). The reason they are unknown is because this ...
- 161
4
votes
0
answers
75
views
Is it a good idea to first train a spiking neural network and then convert it to a conventional neural network?
In many papers about artificial spiking neural networks (SNNs), the performance of them is not up to par with traditional ANNs. I have read how some people have converted ANNs to SNNs using various ...
- 309
4
votes
0
answers
144
views
Is there a mathematical formula that describes the learning curve in neural networks?
In training a neural network, you often see the curve showing how fast the neural network is getting better. It usually grows very fast then slows down to almost horizontal.
Is there a mathematical ...
- 2,216
4
votes
0
answers
82
views
How does the memory mechanism (reading and writing) work in a neural Turing machine?
In neural Turing machine (NTM), reading memory is represented as
\begin{align}
r_t \leftarrow \sum\limits_i^R w_t(i) \mathcal{M}_t(i) \tag{2}
\end{align}
and writing to memory is represented as
...
- 1,076
4
votes
1
answer
590
views
What are the differences between Bytenet and Wavenet?
I recently read Bytenet and Wavenet and I was curious why the first model is not as popular as the second. From my understanding, Bytenet can be seen as a seq2seq model where the encoder and the ...
- 1,148
4
votes
0
answers
123
views
Why isn't the evolutionary Turing machine mainstream?
Given that recurrent neural networks are equivalent to a Turing machine, then why isn't the evolutionary Turing machine, e.g. described in the paper Evolution of evolution: Self-constructing ...
- 331
4
votes
0
answers
602
views
What is the difference between GAT and GaAN?
I was looking at two papers
Graph Attention Networks (GAT) by Petar Veličković and
GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphs by Jiani Zhang.
I'm trying to ...
- 1,148
4
votes
0
answers
148
views
What are stable ways of doing online machine learning?
I am trying to deploy a machine learning solution online into an application for a client. One thing they requested is that the solution must be able to learn online because the problem may be non-...
- 433
4
votes
0
answers
116
views
How do the relative number of cells between neighboring stacked LSTM layers affect the network's behavior?
It seems that stacking LSTM layers can be beneficial for some problem settings in order to learn higher levels of abstraction of temporal relationships in the data. There is already some discussion on ...
- 281
4
votes
0
answers
53
views
Can there be applications of byzantine neural networks on quantum computers?
This question came after I connected 2 pieces of information :
I recently listened to The Byzantine Generals’ Problem, Poisoning, and Distributed Machine Learning with El Mahdi El Mhamdi (Beneficial ...
- 141
4
votes
0
answers
221
views
What characteristics make it difficult for a Neural Network to approximate a function?
What are the characteristics which make a function difficult for the Neural Network to approximate?
Intuitively, one might think uneven functions might be difficult to approximate, but uneven ...
user9947
4
votes
0
answers
129
views
Variable sized input-Multi Label Classification with Neural Network
I have a data input vector ( No Image classification) which size varys from 2 to 7 entrys. Every one of them belongs to a class Out of 7. So I have a variable Input size and a variable Output size. ...
- 51
4
votes
0
answers
1k
views
Can we combine multiple different neural networks in one?
I want to make a kind of robotic brain, i.e. a big neural network, which includes an NLP model (for understanding human voice), real-time object recognition system (so that it can identify particular ...
- 41
4
votes
2
answers
2k
views
How to perform gradient checking in a neural network with batch normalization?
I have implemented a neural network (NN) using python and numpy only for learning purposes. I have already coded learning rate, momentum, and L1/L2 regularization and checked the implementation with ...
4
votes
0
answers
3k
views
Is this a good way to represent Connect 4 to a Neural Network?
I'm attempting to make a bot for the Connect 4 competition on http://riddles.io
My bot isn't horrible, like it's getting up the ladder, but it cannot compete with the winning bots.
I'm using a ...
- 220
4
votes
0
answers
159
views
Train, Validation and Test Split for Reporting Accuracy of Neural Model and BOW
I need to report the accuracies of my neural model in a conference paper as compared to various baselines. What are the accepted standards for reporting accuracies in a fair manner?
Neural Model:
To ...
- 49
4
votes
0
answers
920
views
Can neural networks be used to study (elementary) number theoretic problems?
Can neural networks be used to study (elementary) number theoretic problems? What are examples where this has been done in the past? Or is there on the contrary an understanding that neural networks ...
- 149
3
votes
0
answers
92
views
Is improving a Neural Network really just "trial and error"?
After asking on StackOverflow, I was redirected here, so I'm reposting this question.
I am a PhD student in Computational Physics and I've started to study a bit of Neural Networks, and decided to try ...
- 131
3
votes
0
answers
161
views
How do I balance context and history when creating prompts for LLM's?
A conversation through the OpenAI API looks something like this
...
- 131
3
votes
0
answers
497
views
Relation between SDE diffusion and DDPM/DDIM
Background & Definitions
In DDPM, the diffusion backward step is described as follows (where $z\sim \mathcal{N}(0,I)$ and $x_{T}\sim \mathcal{N}(0,I)$):
and in DDIM we have
while in the SDE ...
- 79
3
votes
0
answers
205
views
Best calculus books for Deep Learning
Recommend some calculus books for Deep Learning and neural networks. I know what is integration, differentiation, derivates, limits on a based level. I would like to understand on deep level the ...
- 31
3
votes
0
answers
134
views
How can i tinker my neural network to learn stronger on rare events?
I am training a neural network on a regression problem.
Most of the time the actual y (label) has the same value (say ~0.2) and only in rare cases the actual y is very different (say 2.0 or -2.0)
...
- 31
3
votes
0
answers
616
views
Is there a literature on the time complexity of Neural Networks?
There exist various blog posts describing the time complexity of Fully Connected Neural Networks (1, 2, 3, 4); Convolutional Neural Networks (CNN) (5) and of Long Short-Term Memory (LSTM) networks (6)....
- 31
3
votes
0
answers
304
views
Are there neural networks with (hard) constraints on the weights?
I don't know too much about Deep Learning, so my question might be silly. However, I was wondering whether there are NN architectures with some hard constraints on the weights of some layers. For ...
- 183
3
votes
0
answers
184
views
Which algorithms are used to locate objects in a 3d space?
I can see mobile apps that can locate a 3D object on a surface with a mobile camera and you can turn around that object.
What is the name of the algorithm(s) that is used for that purpose? Or, is ...
- 257
3
votes
0
answers
118
views
How are partial derivatives calculated in a computational graph?
I am trying to understand how are partial derivatives calculated in a computational graph. I understand reasoning behind computational graphs and I am bold enough to say I understand how they work, at ...
- 131
3
votes
0
answers
529
views
Is there any point in adding the position embedding to the class token in Transformers?
The popular implementations of ViTs by Ross Wightman and Phil Wang add the position embedding to the class tokens as well as to the patches.
Is there any point in doing so?
The purpose of introduction ...
- 2,724
3
votes
0
answers
68
views
How does the paper implement NEAT without a global set tracking Innovations?
I have been reading this paper on NEAT and trying to implement the algorithm in C#. For the most part, I understand everything in the paper however, there are 2 things I don't understand that confuse ...
- 51