Questions tagged [multilayer-perceptrons]
For question about Multi Layer Perceptron model/architecture, its training and other related details and parameters associated with the model.
79
questions
0
votes
2
answers
48
views
How are hidden layers counted / semantically defined?
I'm working my way through how LLMs work and I understand how things work but it's not clear to me exactly what is semantically defined as a "layer".
Using the following FFN as an example:
...
- 101
0
votes
1
answer
52
views
Is it possible training accuracy never changed while training?
Question summary
What informations can get from this epoch_accuracy graph?
Is it possible training accuracy never changed like after 10 epoch in graph while training?
Body
I do some experiments with ...
- 27
0
votes
0
answers
22
views
Permute inputs of neural network and expect different outputs?
My question is about if it makes sense to define a function (MLP) that takes two feature vectors f1 and f2. However, I want MLP(f1, f2) != MLP(f2, f1). I believe ...
- 1
0
votes
0
answers
44
views
Why does my loss function fluctuate so much?
I have a loss function that I'm trying to maximise using a neural network.
While it does appear to increase and plateau over the training, it does so in a very "noisy" manner, spiking up and ...
- 73
0
votes
1
answer
25
views
Can a concept/feature be represented using more than one layer of a Neural Network?
I was reading Goodfellow. At the start of the text it was mentioned that there are two ways to represent depth of a deep neural network. One is using the depth of the computation graph and the other ...
- 101
2
votes
1
answer
152
views
Calculating mutual information between layer outputs and targets in a neural network
I've seen in several papers that it is possible to calculate the mutual information between a layer's outputs and the desired outputs. For example:
Source: https://www.ncbi.nlm.nih.gov/pmc/articles/...
- 73
1
vote
0
answers
36
views
Why does averaging attention-weighted positions reduce the effective resolution in transformers?
I was reading this blog post from Harvard and it says in its background paragraph about transformers that the number of operations required to relate signals from two arbitrary input or output ...
- 573
0
votes
0
answers
54
views
How to generate original training videos based on existing videoset?
I am a software engineer who is quickly ramping up on AI tech, but am nevertheless very new to the sector.
A collegue has an extensive collection of training videos, the vertical is wheelchair seating ...
3
votes
1
answer
252
views
Are transformer models better than comparable-complexity MLP-based models?
I've watched the outstanding Andrej Karpathy's From Zero to Hero course. In the last lecture, he introduces Transformer decoder architecture, which is able to produce Shakespear-like text. However, ...
- 131
1
vote
0
answers
108
views
How can an MLP be implemented with convolutional layers?
I am studying the architecture of the network pointnet, specifically the MLPs stages of the pipeline highlighted in red in the following image (taken from the author page here):
It is strange to find ...
3
votes
0
answers
50
views
Why does a neural network struggle to solve this simple problem?
Consider the following problem:
Given a vector x of size dim with values between 0 and 1 (exclusive), determine if ...
- 191
0
votes
0
answers
43
views
Are autoencoders computationally cheaper than MLPs with the same number of neurons?
Are autoencoders computationally cheaper than other neural networks such as MLP with the same number of neurons?
I have read in some papers that autoencoders train the network faster, and I could ...
- 1
2
votes
2
answers
141
views
Why is a simple regression problem so hard for an MLP to learn?
Consider a very simple problem, which is to find the maximum value out of a list of 5 numbers between 0 and 1. This is obviously trivial, but serves as a good example for a real-world problem I'm ...
- 191
3
votes
1
answer
114
views
Why is automatic differentiation still used, if today's computers can calculate symbolic derivatives quite fast?
Today's computers can calculate symbolic derivatives quite fast, why is automatic differentiation still used? For example, Mathematica can handle algebraic operations with arrays. Doesn't automatic ...
- 33
2
votes
1
answer
117
views
Multi-objective training involving maximization of one loss function and minimization of another
I need my model to predict $s$ from my data $x$. Additionally, I need the model to not use signals in $x$ that are predictive of a separate target $a$. My approach is to transform $x$ into a ...
- 121
0
votes
2
answers
241
views
Concise and mathematically-oriented book on AI and neural networks suitable as a gift [closed]
I would like to buy a book about AI and neural networks written on accessible level for a 17 years old mathematically very gifted student interested in these topics. The book should contain some ...
- 67
1
vote
2
answers
59
views
What are all the possible usages of 'multilayer perceptron'?
The term 'multilayer perceptron' has been used in literature in various ways in the literature.
I am presenting some of them below
As a feed-forward neural network [1].
As a fully connected feed-...
- 3,612
2
votes
1
answer
69
views
How to decode P bits that represent a random weight generator?
So I've been tasked by my neural network professor at university to replicate the following research: Intelligent Breast Cancer Diagnosis Using Hybrid GA-ANN.
Each chromosome represents a possible net,...
- 21
0
votes
1
answer
27
views
Do Quo et al (2013) perform backpropagation between layers?
Le et al. 2013's non-weight sharing CNN has inspired me to ask two questions on this site previously.
When training the three-layer autoencoder, do they compute dL/dW (where L is equation 1) ...
- 161
0
votes
1
answer
353
views
How to make a proper approximation of Sine function with Neural Networks?
TL;DR;
How to build a neural network that properly approximates the sine function with different ranges?
Context and Question:
From this question I decided to use the Sergey's answer, however I used a ...
- 111
1
vote
1
answer
82
views
Does a second-order fully-connected layer have any uses?
I was thinking about implementing second-order regression via a fully-connected layer, and I came up with this:
$X$ is the input data, shaped $(features, batch\_number)$.
$w0$ is the bias, shaped $(...
- 113
1
vote
1
answer
83
views
Why is training all layers at a time effective for a multi-layer autoencoder?
This training of all layers of a CNN simultaneously is standard practice today. It is found in every CNN (AlexNet (2012), VGG, Inception, GANs, etc) and even pre-CNN networks such as Le et al. 2012.
...
- 161
1
vote
0
answers
69
views
Is the capability of RNN more than the capability of MLP?
Consider the following excerpt paragraph taken from the section titled "Recurrent Neural Networks" of the chapter 10: Sequence Modeling: Recurrent and Recursive Nets of the textbook named ...
- 3,612
1
vote
0
answers
67
views
Does Godel's incompleteness theorems restricts the scope of connectionist-AI?
It is well-known that Godel's incompleteness theorems restricted the reachability of symbolic-AI, which is dependent on mathematical logic.
But, I am wondering whether it has any impact on the ...
- 3,612
1
vote
1
answer
160
views
Are the capabilities of connectionist AI and symbolic AI the same?
The universal approximation theorem says that MLP with a single hidden layer and enough number of neurons can able to approximate any bounded continuous function. You can validate it from the ...
- 3,612
1
vote
1
answer
69
views
Rank of gradient-of-loss with respect to layer weights in an MLP
The paper: https://arxiv.org/abs/2110.11309, makes the following claim at the end of page 3:
The gradient of loss $L$ with respect to weights $W_l$ of an MLP is a rank-1 matrix for each of B batch ...
- 63
1
vote
0
answers
66
views
Is the VC dimension of a MLP regressor a valid upper bound on how many points it can exactly fit?
I want to calculate an upper bound on how many training points an MLP regressor can fit with ~0 error. I don't care about the test error, I want to overfit as much as possible the (few) training ...
- 11
1
vote
1
answer
246
views
When should we use CNN instead of MLP?
Is CNN only applicable to time-series data or image data?
When should we use CNN instead of MLP?
- 299
0
votes
1
answer
2k
views
What, exactly, do mlp(64,64) and mlp(64,128,1024) mean in PointNet, and how many input neurons does 1 (x,y,z) point have?
I couldn't find out how to interpret the multilayer perceptron notation given in PointNet. Specifically, I am looking to find out what the numbers inside the parentheses of ...
- 41
1
vote
1
answer
486
views
Why doesn't the LSTM model improve the time-series forecasting significantly with respect to the MLP model?
I have recently started learning time series forecasting. I have a dataset of the weekly payment history of 10k clients over 1 year, and I want to predict the future 5 payments for a test set of 1k ...
- 111
0
votes
1
answer
105
views
About the choice of the activation functions in the Multilayer Perceptron, and on what does this depends?
I've read in this: F. Rosenblatt, Principles of neurodynamics. perceptrons and the theory of brain mechanisms that in the Multilayer Perceptron the activation functions in the second, third, ..., are ...
0
votes
0
answers
65
views
What are the math theorems regarding the Multilayer Perceptron?
I've come across a theorem "Convergence theorem
Simple Perceptron" for the first time, here-> https://zaguan.unizar.es/record/69205/files/TAZ-TFG-2018-148.pdf, page 27, (is in Spanish)
...
2
votes
1
answer
404
views
Why is the backpropagation algorithm used to train the multilayer perceptron?
I've read in the book Neural Network Design, by Martin Hagan et al. (chapter 11), that, to train the feed-forward neural network (aka multilayer perceptron), one uses the backpropagation algorithm.
...
0
votes
0
answers
163
views
What is the difference between the forward pass of the Multi-Layer Perceptron, Deep AutoEncoder and Deep Belief Network?
Multi-Layer Perceptron (MLP), Deep AutoEncoder (DAE), and Deep Belief Network (DBN) are trained differently.
However, do they follow the same process during the inference phase, i.e., do they ...
- 73
1
vote
1
answer
79
views
Can RNNs get inputs and produce outputs similar to the inputs and outputs of FFNNs?
RNN and LSTM models have many architectures that can be modified. We can also compose their input and output data. However, in the examples that I found on the web, the inputs and outputs of RNNs/...
- 45
1
vote
1
answer
453
views
What are examples of good free books that cover the back-propagation algorithm?
What are examples of good free books that cover the back-propagation used to train multilayer perceptrons? I've just started to learn about artificial neural networks, so I'm looking for books that ...
6
votes
1
answer
1k
views
Why do feedforward neural networks require the inputs to be of a fixed size, while RNNs can process variable-size inputs?
Why does a vanilla feedforward neural network only accept a fixed input size, while RNNs are capable of taking a series of inputs with no predetermined limit on the size? Can anyone elaborate on this ...
- 63
19
votes
1
answer
879
views
What is the number of neurons required to approximate a polynomial of degree n?
I learned about the universal approximation theorem from this guide. It states that a network even with a single hidden layer can approximate any function within some bound, given a sufficient number ...
- 763
0
votes
0
answers
80
views
How to draw a 3-dimensonal shape's neural network
I am reading an exam question about NN (that I cannot publish, for copyright reasons). The question says: 'Construct a rectangle in 2D space. Define the lines, and then define the weights and ...
- 297
2
votes
2
answers
813
views
Why don't neural networks project the data into higher dimensions first, then reduce the size of each layer thereafter?
Background
From my understanding (and following along with this blog post), (deep) neural networks apply transformations to the data such that the data's representation to the next layer (or ...
- 133
0
votes
1
answer
711
views
Keras DQN Model with Multiple Inputs and Multiple Outputs [closed]
I am trying to create a DQN agent where I have 2 inputs: the agent's position and a matrix of 0s and 1s.
The output is composed of the agent's new chosen position, a matrix of 0s and 1s (different ...
- 206
3
votes
1
answer
79
views
Is there a common way to build a neural network that seeks to extract spatial and temporal information simultaneously?
Is there a common way to build a neural network that seeks to extract spatial and temporal information simultaneously? Is there an agreed up protocol on how to extract this information?
What ...
- 165
1
vote
2
answers
1k
views
How is the error calculated with multiple output neurons in the neural network?
Machine Learning books generally explains that the error calculated for a given sample $i$ is:
$e_i = y_i - \hat{y_i}$
Where $\hat{y}$ is the target output and $y$ is the actual output given by the ...
- 111
3
votes
1
answer
518
views
Why does every neuron in hidden layers of a multi-layer perceptron typically have the same activation function? [duplicate]
Why does every neuron in a hidden layer of a multi-layer perceptron (MLP) typically have the same activation function as every other neuron in the same or other hidden layers (so I exclude the output ...
- 737
1
vote
1
answer
3k
views
Why can't MLPs perform non-linear regression and classification?
In this page it's told:
In Single Perceptron / Multi-layer Perceptron(MLP), we only have linear separability because they are composed of input and output layers(some hidden layers in MLP)
What ...
- 167
2
votes
2
answers
211
views
What are standard datasets for fully connected neural networks?
I am looking for datasets that are used as a testing standard in the fully connected neural networks (FCNN). For example, in the image recognition and CNN, CIFAR datasets are used in most of the ...
- 333
5
votes
2
answers
1k
views
Why is it called back-propagation?
While looking at the mathematics of the back-propagation algorithm for a multi-layer perceptron, I noticed that in order to find the partial derivative of the cost function with respect to a weight (...
- 153
6
votes
2
answers
1k
views
Can neurons in MLP and filters in CNN be compared?
I know they are not the same in working, but an input layer sends the input to $n$ neurons with a set of weights, based on these weights and the activation layer, it produces an output that can be fed ...
- 193
4
votes
1
answer
148
views
Why MLP cannot approximate a closed shape function?
[TL;DR]
I generated two classes Red and Blue on a 2D space. Red are points on Unit Circle and Blue are points on a Circle Ring with radius limits (3,4). I tried to train a Multi Layer Perceptron ...
- 143
2
votes
1
answer
99
views
Recent algorithms for correcting mislabeled data using multilayer perceptrons
I am doing literature research on algorithms for correcting mislabeled data using multilayer perceptrons. Found an "old" paper An algorithm for correcting mislabeled data (2001) by Xinchuan Zeng et al....
- 63