Questions tagged [neural-networks]

For questions about a artificial networks, such as MLPs, CNNs, RNNs, LSTM, and GRU networks, their variants or any other AI system components that qualify as a neural networks in that they are, in part, inspired by biological neural networks.

857 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
10 votes
1 answer
905 views

Are Cellular Neural Networks one type of Neural Networks?

I am researching Cellular Neural Networks and have already read Chua's two articles (1988). In cellular neural networks, a cell is only in relation with its neighbors. So it is easy to use them for ...
netizen's user avatar
  • 283
8 votes
1 answer
168 views

What is the impact of using multiple BMUs for self-organizing maps?

Here's a sort of a conceptual question. I was implementing a SOM algorithm to better understand its variations and parameters. I got curious about one bit: the BMU (best matching unit == the neuron ...
Rafael Santos's user avatar
7 votes
1 answer
146 views

How does the network know which objects to track in the paper "Label-Free Supervision of Neural Networks with Physics and Domain Knowledge"?

I was reading the paper Label-Free Supervision of Neural Networks with Physics and Domain Knowledge, published at AAAI 2017, which won the best paper award. I understand the math and it makes sense. ...
sanjeev mk's user avatar
6 votes
1 answer
102 views

It is possible to use deep learning to give approximate solutions to NP-hard graph theory problems?

It is possible to use deep learning to give approximate solutions to NP-hard graph theory problems? If we take, for example, the travelling salesman problem (or the dominating set problem). Let's say ...
Jake B.'s user avatar
  • 181
6 votes
1 answer
177 views

How to classify human actions?

I'm quite new to machine learning (I followed the Coursera course of Andrew Ng and now starting deeplearning.ai courses). I want to classify human actions real-time like: Left-arm bended Arm above ...
user1007522's user avatar
6 votes
1 answer
212 views

How do big companies, like Facebook, model individuals and their interaction?

As a layman in AI, I want to get an idea of how big data players, like Facebook, model individuals (of which they have so many data). There are two scenarios I can imagine: Neural networks build ...
Hans-Peter Stricker's user avatar
5 votes
1 answer
2k views

Which other loss functions for hierarchical multi-label classification could I use?

I am looking to try different loss functions for a hierarchical multi-label classification problem. So far, I have been training different models or submodels like multilayer perceptron (MLP) branch ...
Skinish's user avatar
  • 163
4 votes
0 answers
655 views

What is a neural network compatibility function?

Typically, a neural network parameterized by weights $\mathbf{W}$ is a function from an input $x$ to an output $y$. The network has an associated compatibility function $\Psi(y; x, \mathbf{W}) \...
Yashas's user avatar
  • 189
4 votes
2 answers
218 views

Which neural network can I use to solve this constrained optimisation problem?

Let $\mathcal{S}$ be the training data set, where each input $u^i \in \mathcal{S}$ has $d$ features. I want to design an ANN so that the cost function below is minimized (the sum of the square of ...
user3489173's user avatar
4 votes
0 answers
65 views

Can the quality of randomness in neural network initialization affect model fitting?

This is a topic I have been arguing about for some time now with my colleagues, maybe you could also voice your opinion about it. Artificial neural networks use random weight initialization within a ...
Aki Koivu's user avatar
4 votes
0 answers
86 views

Visualizing the Loss Landscape of Neural Nets: Meaning of the word 'filter'?

I found myself scratching my head when I read the following phrase in the paper Visualizing the Loss Landscape of Neural Nets: To remove this scaling effect, we plot loss functions using filter-wise ...
hijit's user avatar
  • 41
4 votes
0 answers
163 views

Why does sigmoid saturation prevent signal flow through the neuron?

As per these slides on page 35: Sigmoids saturate and kill gradients. when the neuron's activation saturates at either tail of 0 or 1, the gradient at these regions is almost zero. the gradient and ...
EEAH's user avatar
  • 193
4 votes
0 answers
244 views

Why are most commonly used activation functions continuous?

I have come to notice that the most commonly used activation functions are continuous. Is there any specific reason behind this? Results such as this paper have worked on training networks with ...
ABIM's user avatar
  • 545
4 votes
0 answers
80 views

When do two identical neural networks have uncorrelated errors?

In Chapter 9, section 9.1.6, Raul Rojas describes how committees of networks can reduce the prediction error by training N identical neural networks and averaging the results. If $f_i$ are the ...
EmmanuelMess's user avatar
4 votes
0 answers
65 views

When is using weight regularization bad?

Regularization of weights (e.g. L1 or L2) keeps them small and standardized, which can help reduce data overfitting. From this article, regularization sounds favorable in many cases, but is it always ...
mark mark's user avatar
  • 763
4 votes
0 answers
191 views

Has the logistic map ever been used as an activation function?

I find the logistic map absolutely fascinating. Both in itself (because I love fractal) and because it is observed in nature (see: https://www.youtube.com/watch?v=ovJcsL7vyrk). I'm wondering if anyone ...
ker2x's user avatar
  • 163
4 votes
0 answers
2k views

Should batch normalisation be applied before or after ReLU?

I know that there has been some discussion about this (e.g. here and here), but I can't seem to find consensus. The crucial thing that I haven't seen mentioned in these discussions is that applying ...
Kris's user avatar
  • 171
4 votes
0 answers
274 views

Understanding the results of "Visualizing and Understanding Convolutional Networks"

I am trying to understand the results of the paper Visualizing and Understanding Convolutional Networks, in particular the following image: What are these 3x3 blocks and their 9 cells representing? ...
Andreas K.'s user avatar
4 votes
0 answers
39 views

How do weights changes handles during back-propagation when there are unknown labels

I have a question about how weights are updated during back-propagation for some of my samples that have unknown labels (please note, unknown, not missing). The reason they are unknown is because this ...
user9317212's user avatar
4 votes
0 answers
70 views

Is it a good idea to first train a spiking neural network and then convert it to a conventional neural network?

In many papers about artificial spiking neural networks (SNNs), the performance of them is not up to par with traditional ANNs. I have read how some people have converted ANNs to SNNs using various ...
KaneM's user avatar
  • 309
4 votes
0 answers
135 views

Is there a mathematical formula that describes the learning curve in neural networks?

In training a neural network, you often see the curve showing how fast the neural network is getting better. It usually grows very fast then slows down to almost horizontal. Is there a mathematical ...
zooby's user avatar
  • 2,196
4 votes
0 answers
80 views

How does the memory mechanism (reading and writing) work in a neural Turing machine?

In neural Turing machine (NTM), reading memory is represented as \begin{align} r_t \leftarrow \sum\limits_i^R w_t(i) \mathcal{M}_t(i) \tag{2} \end{align} and writing to memory is represented as ...
Eka's user avatar
  • 1,066
4 votes
1 answer
548 views

What are the differences between Bytenet and Wavenet?

I recently read Bytenet and Wavenet and I was curious why the first model is not as popular as the second. From my understanding, Bytenet can be seen as a seq2seq model where the encoder and the ...
razvanc92's user avatar
  • 1,128
4 votes
0 answers
120 views

Why isn't the evolutionary Turing machine mainstream?

Given that recurrent neural networks are equivalent to a Turing machine, then why isn't the evolutionary Turing machine, e.g. described in the paper Evolution of evolution: Self-constructing ...
Dimer's user avatar
  • 331
4 votes
0 answers
586 views

What is the difference between GAT and GaAN?

I was looking at two papers Graph Attention Networks (GAT) by Petar Veličković and GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphs by Jiani Zhang. I'm trying to ...
razvanc92's user avatar
  • 1,128
4 votes
0 answers
143 views

What are stable ways of doing online machine learning?

I am trying to deploy a machine learning solution online into an application for a client. One thing they requested is that the solution must be able to learn online because the problem may be non-...
Rui Nian's user avatar
  • 433
4 votes
0 answers
115 views

How do the relative number of cells between neighboring stacked LSTM layers affect the network's behavior?

It seems that stacking LSTM layers can be beneficial for some problem settings in order to learn higher levels of abstraction of temporal relationships in the data. There is already some discussion on ...
adamconkey's user avatar
4 votes
0 answers
46 views

Can there be applications of byzantine neural networks on quantum computers?

This question came after I connected 2 pieces of information : I recently listened to The Byzantine Generals’ Problem, Poisoning, and Distributed Machine Learning with El Mahdi El Mhamdi (Beneficial ...
Paul Rougieux's user avatar
4 votes
0 answers
212 views

What characteristics make it difficult for a Neural Network to approximate a function?

What are the characteristics which make a function difficult for the Neural Network to approximate? Intuitively, one might think uneven functions might be difficult to approximate, but uneven ...
user avatar
4 votes
2 answers
2k views

How to perform gradient checking in a neural network with batch normalization?

I have implemented a neural network (NN) using python and numpy only for learning purposes. I have already coded learning rate, momentum, and L1/L2 regularization and checked the implementation with ...
Arnaldo Gualberto's user avatar
4 votes
4 answers
1k views

Use Machine/Deep Learning to Guess a String

I want to be able to input a block of text and then have it guess a string within a predefined range (i.e. a string that starts with three letters and ends with five numbers like "XXX12345", etc). ...
TreHoffman's user avatar
4 votes
0 answers
3k views

Is this a good way to represent Connect 4 to a Neural Network?

I'm attempting to make a bot for the Connect 4 competition on http://riddles.io My bot isn't horrible, like it's getting up the ladder, but it cannot compete with the winning bots. I'm using a ...
NeomerArcana's user avatar
4 votes
0 answers
916 views

Can neural networks be used to study (elementary) number theoretic problems?

Can neural networks be used to study (elementary) number theoretic problems? What are examples where this has been done in the past? Or is there on the contrary an understanding that neural networks ...
Stefan Witzel's user avatar
3 votes
0 answers
91 views

Is improving a Neural Network really just "trial and error"?

After asking on StackOverflow, I was redirected here, so I'm reposting this question. I am a PhD student in Computational Physics and I've started to study a bit of Neural Networks, and decided to try ...
Mauro Giliberti's user avatar
3 votes
0 answers
245 views

Relation between SDE diffusion and DDPM/DDIM

Background & Definitions In DDPM, the diffusion backward step is described as follows (where $z\sim \mathcal{N}(0,I)$ and $x_{T}\sim \mathcal{N}(0,I)$): and in DDIM we have while in the SDE ...
snatchysquid's user avatar
3 votes
0 answers
110 views

how do you train a neural network to determine shortest path in a 4-node graph

Suppose I have the following graph: ...
user1068636's user avatar
3 votes
0 answers
162 views

Best calculus books for Deep Learning

Recommend some calculus books for Deep Learning and neural networks. I know what is integration, differentiation, derivates, limits on a based level. I would like to understand on deep level the ...
Dan Il's user avatar
  • 31
3 votes
0 answers
93 views

How can i tinker my neural network to learn stronger on rare events?

I am training a neural network on a regression problem. Most of the time the actual y (label) has the same value (say ~0.2) and only in rare cases the actual y is very different (say 2.0 or -2.0) ...
Carl Philip's user avatar
3 votes
0 answers
55 views

Why does a neural network struggle to solve this simple problem?

Consider the following problem: Given a vector x of size dim with values between 0 and 1 (exclusive), determine if ...
Daniel's user avatar
  • 191
3 votes
0 answers
271 views

Are there neural networks with (hard) constraints on the weights?

I don't know too much about Deep Learning, so my question might be silly. However, I was wondering whether there are NN architectures with some hard constraints on the weights of some layers. For ...
Onil90's user avatar
  • 183
3 votes
0 answers
157 views

Which algorithms are used to locate objects in a 3d space?

I can see mobile apps that can locate a 3D object on a surface with a mobile camera and you can turn around that object. What is the name of the algorithm(s) that is used for that purpose? Or, is ...
dasmehdix's user avatar
  • 257
3 votes
0 answers
101 views

How are partial derivatives calculated in a computational graph?

I am trying to understand how are partial derivatives calculated in a computational graph. I understand reasoning behind computational graphs and I am bold enough to say I understand how they work, at ...
dosvarog's user avatar
  • 131
3 votes
1 answer
157 views

Is there a way to update the neural network to fit the new data without the time required for retraining?

I built a basic neural network in MATLAB. The neural network classifies points on the X-Y axis system into two classes (0 and 1). (I try to get the function that represents a shape from this photo) ...
shlomo odem's user avatar
3 votes
0 answers
409 views

Is there any point in adding the position embedding to the class token in Transformers?

The popular implementations of ViTs by Ross Wightman and Phil Wang add the position embedding to the class tokens as well as to the patches. Is there any point in doing so? The purpose of introduction ...
spiridon_the_sun_rotator's user avatar
3 votes
0 answers
67 views

How does the paper implement NEAT without a global set tracking Innovations?

I have been reading this paper on NEAT and trying to implement the algorithm in C#. For the most part, I understand everything in the paper however, there are 2 things I don't understand that confuse ...
Steve Forbes's user avatar
3 votes
0 answers
404 views

Loss function to minimize the distance between sets

Are there references or links to examples about loss functions "Distance Metrics" which could be used to minimize the distance between two sets for a neural network. More precisely, this ...
Noah16's user avatar
  • 131
3 votes
0 answers
319 views

How do I implement the cross-entropy-method for a RL environment with a continuous action space?

I found many tutorials and posts on how to solve RL environments with discrete action spaces using the cross entropy method (e.g., in this blog post for the OpenAI Gym frozen lake environment). ...
Philipp's user avatar
  • 143
3 votes
1 answer
184 views

What is asymmetric relaxation backpropagation?

In Chapter 8, section 8.5.2, Raul Rojas describes how the weights for a layer of a neural network can be calculated using a pseudoinverse of the sigmoid function in the nodes, he explains this is an ...
EmmanuelMess's user avatar
3 votes
0 answers
292 views

Image classification - Need method to classify "unknown" objects as "trash" (3D objects)

We have an image classifier that was built using CNN with faster R-CNN and Yolov5. It is designated to run on 3D objects. All of those objects have similar "features" structure, but the ...
Stackaloo's user avatar
3 votes
0 answers
71 views

How efficient is SCAWI weight initialization method?

I'm currently in the middle of a project (for my thesis) constructing a deep neural network. Since I'm still in the research part, I'm trying to find various ways and techniques to initialize weights. ...
ChrisP's user avatar
  • 41

1
2 3 4 5
18