Questions tagged [multilayer-perceptrons]
For question about Multi Layer Perceptron model/architecture, its training and other related details and parameters associated with the model.
95 questions
1
vote
1
answer
39
views
Gradient calculation in Backpropogation
Some notations for the question: $w_{ij}^l$ is the weight connecting ith neuron of the layer l to the jth neuron of the layer $l-1$. $z_i^l$ is the activation of ith neuron in the layer l (for ...
- 13
0
votes
0
answers
30
views
Simple neural network applications
I am wondering if there are some applications for simple CNNs and MLPs that are preferred to complex deep neural networks. For example, tabular data is still working with traditional ML algorithms and ...
- 1
0
votes
0
answers
25
views
Understanding a formula in Rosenblatt's perceptron paper
I want to be an AI researcher and realized that I've never really read an AI paper (or any academic paper for that matter) all the way through, and seriously tried to understand the content. Before ...
- 101
1
vote
1
answer
46
views
MLP Gaussian Decoder in VAE
My question concerns the paper arxiv.org/pdf/1312.6114. I want to know why they proposed to use MLP Gaussian decoder with parameters given by the MLP transformation of the z variable as the likelihood ...
- 133
0
votes
1
answer
31
views
How can you solve a machine-learning linear-regression problem for a curvilinear relationship?
I'm aghast at how difficult of a problem it is for me to solve the function f(x) = x^2 with a linear-regression multi-layer-perceptron approach with PyTorch.
I'm ...
0
votes
1
answer
83
views
Using conditional probability as an estimate in a loss function
I have a rather large ML framework that takes multiple conditional probability terms that are computed via classifiers/neural networks. This arbitrary loss function is computed via a function:
...
- 101
0
votes
0
answers
101
views
Should batch normalization be used before or after, when using sigmoid or tanh?
This prior post discusses Batch-norm with Relu ordering.
I have a similar question but pertaining to sigmoid or tanh kind of activation functions. Should batch-norm be used before or after? Is there a ...
- 1
1
vote
0
answers
88
views
Why doesn't the Kolmogorov-Arnold representation theorem imply an MLP-like structure?
Recently, Kolmogorov-Arnold Networks (KANs) generated a lot of hype, with "AI experts" throwing around terms like "ML 2.0" and "a new era of ML".
KANs are supposedly ...
- 11
1
vote
0
answers
34
views
How a MLP learns from images and boundaries of its capabilities
(Sorry, I'm not very experienced with ML and I apologize if these questions are vague, naive, or not written well. I also moved this from CV to here.)
I think I understand that a MLP learns (via ...
- 11
0
votes
1
answer
28
views
Proof that Temporal-Difference TD(1) is Equivalent to Widrow-Hoff
I'm reading Sutton's "Learning to Predict by the Methods of Temporal Differences" and I'm getting hung up on a derivation (p. 14).
We are considering (observation-sequence, outcome) pairs. $[...
0
votes
0
answers
27
views
Neural Network with Incorrect Calculation Better than Correct One
I have designed my own neural network and discovered an error. During backpropagation, instead of inserting the Z-values into the derivative of the activation function, I inserted the A-values. The ...
- 1
0
votes
1
answer
110
views
Xavier vs He initialization with tanh
I'm a student and in the lecture, I learned that He initialization is better than Xavier if you use ReLU activation function.
In addition, I also learned that Xavier initialization is better than He ...
- 13
0
votes
1
answer
116
views
Total loss in backpropagation
I'd say I have some understanding of backpropagation, however I am not really sure of the total loss being calculated.
Let us take the example below:
After 1 forward pass when I have to update the ...
- 9
0
votes
1
answer
50
views
How to optimise a FNN/MLP network with MSE (positive only loss) in C
I can create a FNN/MLP network in C but only g-p loss works, where g = ground truth and p = predicted.
What I don't understand is how MSE a positive only loss value can train a back propagation ...
user81637
2
votes
1
answer
106
views
Is it a requirement/recommendation to normalize my inputs into [0,1] range?
All features of my input dataset, which is going to be used for training a simple multi-layered neural network, are in range $[-1,+1]$ and the output of $NN$ is a single number again in range $[-1,+1]$...
- 23
0
votes
0
answers
8
views
Cannot find appropriate model to classify hidden states
My input data is vector representing encoded image - 22 features, and I try to classify by 3 classes 0, 1, 2 (neutral, good, bad)
Original:
...
- 101
0
votes
1
answer
601
views
Validation loss is always lower than training loss whatever i try
I've been training several types of MLPs with different optimisers and tuned them with keras's hyperband tuner. All of them follow this cone architecture:
All the networks were trained on the same ...
0
votes
2
answers
66
views
How are hidden layers counted / semantically defined?
I'm working my way through how LLMs work and I understand how things work but it's not clear to me exactly what is semantically defined as a "layer".
Using the following FFN as an example:
...
- 213
0
votes
1
answer
61
views
Is it possible training accuracy never changed while training?
Question summary
What informations can get from this epoch_accuracy graph?
Is it possible training accuracy never changed like after 10 epoch in graph while training?
Body
I do some experiments with ...
- 69
0
votes
0
answers
94
views
Why does my loss function fluctuate so much?
I have a loss function that I'm trying to maximise using a neural network.
While it does appear to increase and plateau over the training, it does so in a very "noisy" manner, spiking up and ...
- 83
0
votes
1
answer
29
views
Can a concept/feature be represented using more than one layer of a Neural Network?
I was reading Goodfellow. At the start of the text it was mentioned that there are two ways to represent depth of a deep neural network. One is using the depth of the computation graph and the other ...
- 101
3
votes
1
answer
482
views
Calculating mutual information between layer outputs and targets in a neural network
I've seen in several papers that it is possible to calculate the mutual information between a layer's outputs and the desired outputs. For example:
Source: https://www.ncbi.nlm.nih.gov/pmc/articles/...
- 83
1
vote
1
answer
137
views
Why does averaging attention-weighted positions reduce the effective resolution in transformers?
I was reading this blog post from Harvard and it says in its background paragraph about transformers that the number of operations required to relate signals from two arbitrary input or output ...
- 575
0
votes
0
answers
168
views
How to generate original training videos based on existing videoset?
I am a software engineer who is quickly ramping up on AI tech, but am nevertheless very new to the sector.
A collegue has an extensive collection of training videos, the vertical is wheelchair seating ...
3
votes
1
answer
774
views
Are transformer models better than comparable-complexity MLP-based models?
I've watched the outstanding Andrej Karpathy's From Zero to Hero course. In the last lecture, he introduces Transformer decoder architecture, which is able to produce Shakespear-like text. However, ...
- 131
1
vote
1
answer
296
views
How can an MLP be implemented with convolutional layers?
I am studying the architecture of the network pointnet, specifically the MLPs stages of the pipeline highlighted in red in the following image (taken from the author page here):
It is strange to find ...
4
votes
0
answers
71
views
Why does a neural network struggle to solve this simple problem?
Consider the following problem:
Given a vector x of size dim with values between 0 and 1 (exclusive), determine if ...
- 201
0
votes
0
answers
50
views
Are autoencoders computationally cheaper than MLPs with the same number of neurons?
Are autoencoders computationally cheaper than other neural networks such as MLP with the same number of neurons?
I have read in some papers that autoencoders train the network faster, and I could ...
- 1
2
votes
2
answers
168
views
Why is a simple regression problem so hard for an MLP to learn?
Consider a very simple problem, which is to find the maximum value out of a list of 5 numbers between 0 and 1. This is obviously trivial, but serves as a good example for a real-world problem I'm ...
- 201
3
votes
1
answer
166
views
Why is automatic differentiation still used, if today's computers can calculate symbolic derivatives quite fast?
Today's computers can calculate symbolic derivatives quite fast, why is automatic differentiation still used? For example, Mathematica can handle algebraic operations with arrays. Doesn't automatic ...
- 33
2
votes
1
answer
140
views
Multi-objective training involving maximization of one loss function and minimization of another
I need my model to predict $s$ from my data $x$. Additionally, I need the model to not use signals in $x$ that are predictive of a separate target $a$. My approach is to transform $x$ into a ...
- 121
-1
votes
2
answers
313
views
Concise and mathematically-oriented book on AI and neural networks suitable as a gift [closed]
I would like to buy a book about AI and neural networks written on accessible level for a 17 years old mathematically very gifted student interested in these topics. The book should contain some ...
- 65
1
vote
2
answers
68
views
What are all the possible usages of 'multilayer perceptron'?
The term 'multilayer perceptron' has been used in literature in various ways in the literature.
I am presenting some of them below
As a feed-forward neural network [1].
As a fully connected feed-...
- 3,990
2
votes
1
answer
92
views
How to decode P bits that represent a random weight generator?
So I've been tasked by my neural network professor at university to replicate the following research: Intelligent Breast Cancer Diagnosis Using Hybrid GA-ANN.
Each chromosome represents a possible net,...
- 21
0
votes
1
answer
37
views
Do Quo et al (2013) perform backpropagation between layers?
Le et al. 2013's non-weight sharing CNN has inspired me to ask two questions on this site previously.
When training the three-layer autoencoder, do they compute dL/dW (where L is equation 1) ...
- 181
0
votes
1
answer
481
views
How to make a proper approximation of Sine function with Neural Networks?
TL;DR;
How to build a neural network that properly approximates the sine function with different ranges?
Context and Question:
From this question I decided to use the Sergey's answer, however I used a ...
- 111
1
vote
1
answer
92
views
Does a second-order fully-connected layer have any uses?
I was thinking about implementing second-order regression via a fully-connected layer, and I came up with this:
$X$ is the input data, shaped $(features, batch\_number)$.
$w0$ is the bias, shaped $(...
- 113
1
vote
1
answer
101
views
Why is training all layers at a time effective for a multi-layer autoencoder?
This training of all layers of a CNN simultaneously is standard practice today. It is found in every CNN (AlexNet (2012), VGG, Inception, GANs, etc) and even pre-CNN networks such as Le et al. 2012.
...
- 181
1
vote
0
answers
81
views
Is the capability of RNN more than the capability of MLP?
Consider the following excerpt paragraph taken from the section titled "Recurrent Neural Networks" of the chapter 10: Sequence Modeling: Recurrent and Recursive Nets of the textbook named ...
- 3,990
1
vote
0
answers
83
views
Does Godel's incompleteness theorems restricts the scope of connectionist-AI?
It is well-known that Godel's incompleteness theorems restricted the reachability of symbolic-AI, which is dependent on mathematical logic.
But, I am wondering whether it has any impact on the ...
- 3,990
1
vote
1
answer
269
views
Are the capabilities of connectionist AI and symbolic AI the same?
The universal approximation theorem says that MLP with a single hidden layer and enough number of neurons can able to approximate any bounded continuous function. You can validate it from the ...
- 3,990
3
votes
1
answer
101
views
Rank of gradient-of-loss with respect to layer weights in an MLP
The paper: https://arxiv.org/abs/2110.11309, makes the following claim at the end of page 3:
The gradient of loss $L$ with respect to weights $W_l$ of an MLP is a rank-1 matrix for each of B batch ...
- 141
1
vote
0
answers
78
views
Is the VC dimension of a MLP regressor a valid upper bound on how many points it can exactly fit?
I want to calculate an upper bound on how many training points an MLP regressor can fit with ~0 error. I don't care about the test error, I want to overfit as much as possible the (few) training ...
- 11
1
vote
1
answer
348
views
When should we use CNN instead of MLP?
Is CNN only applicable to time-series data or image data?
When should we use CNN instead of MLP?
- 341
0
votes
1
answer
2k
views
What, exactly, do mlp(64,64) and mlp(64,128,1024) mean in PointNet, and how many input neurons does 1 (x,y,z) point have?
I couldn't find out how to interpret the multilayer perceptron notation given in PointNet. Specifically, I am looking to find out what the numbers inside the parentheses of ...
- 51
1
vote
1
answer
661
views
Why doesn't the LSTM model improve the time-series forecasting significantly with respect to the MLP model?
I have recently started learning time series forecasting. I have a dataset of the weekly payment history of 10k clients over 1 year, and I want to predict the future 5 payments for a test set of 1k ...
- 111
0
votes
1
answer
120
views
About the choice of the activation functions in the Multilayer Perceptron, and on what does this depends?
I've read in this: F. Rosenblatt, Principles of neurodynamics. perceptrons and the theory of brain mechanisms that in the Multilayer Perceptron the activation functions in the second, third, ..., are ...
0
votes
0
answers
77
views
What are the math theorems regarding the Multilayer Perceptron?
I've come across a theorem "Convergence theorem
Simple Perceptron" for the first time, here-> https://zaguan.unizar.es/record/69205/files/TAZ-TFG-2018-148.pdf, page 27, (is in Spanish)
...
2
votes
1
answer
439
views
Why is the backpropagation algorithm used to train the multilayer perceptron?
I've read in the book Neural Network Design, by Martin Hagan et al. (chapter 11), that, to train the feed-forward neural network (aka multilayer perceptron), one uses the backpropagation algorithm.
...
0
votes
0
answers
174
views
What is the difference between the forward pass of the Multi-Layer Perceptron, Deep AutoEncoder and Deep Belief Network?
Multi-Layer Perceptron (MLP), Deep AutoEncoder (DAE), and Deep Belief Network (DBN) are trained differently.
However, do they follow the same process during the inference phase, i.e., do they ...
- 73