Questions tagged [multilayer-perceptrons]

For question about Multi Layer Perceptron model/architecture, its training and other related details and parameters associated with the model.

Filter by
Sorted by
Tagged with
1 vote
1 answer
61 views

Why does averaging attention-weighted positions reduce the effective resolution in transformers?

I was reading this blog post from Harvard and it says in its background paragraph about transformers that the number of operations required to relate signals from two arbitrary input or output ...
2 votes
1 answer
74 views

How to decode P bits that represent a random weight generator?

So I've been tasked by my neural network professor at university to replicate the following research: Intelligent Breast Cancer Diagnosis Using Hybrid GA-ANN. Each chromosome represents a possible net,...
1 vote
1 answer
77 views

Rank of gradient-of-loss with respect to layer weights in an MLP

The paper: https://arxiv.org/abs/2110.11309, makes the following claim at the end of page 3: The gradient of loss $L$ with respect to weights $W_l$ of an MLP is a rank-1 matrix for each of B batch ...
2 votes
1 answer
124 views

Multi-objective training involving maximization of one loss function and minimization of another

I need my model to predict $s$ from my data $x$. Additionally, I need the model to not use signals in $x$ that are predictive of a separate target $a$. My approach is to transform $x$ into a ...
0 votes
0 answers
8 views

Cannot find appropriate model to classify hidden states

My input data is vector representing encoded image - 22 features, and I try to classify by 3 classes 0, 1, 2 (neutral, good, bad) Original: ...
2 votes
2 answers
573 views

Unable to overfit using MLP

I'm building a 5-class classifier with a private dataset. Each data sample has 67 features and there are about 40000 samples. Samples of a particular class were duplicated to overcome class imbalance ...
0 votes
1 answer
27 views

Validation loss is always lower than training loss whatever i try

I've been training several types of MLPs with different optimisers and tuned them with keras's hyperband tuner. All of them follow this cone architecture: All the networks were trained on the same ...
2 votes
1 answer
75 views

Applying Machine Learning to 2D Laser Scanner Data

We are using 2D Laser Scanner to scan various objects of different geometric shapes for e.g. cylinder, spiked, cylinder with notch, cylinder with curved edges e.t.c. The dataset contains points in the ...
3 votes
2 answers
248 views

Which online machine learning technique to use for multi-class classification problem with multiple inputs?

I have the following problem. We have $4$ separate discrete inputs, which can take any integer value between $-63$ and $63$. The output is also supposed to be a discrete value between $-63$ and $63$. ...
0 votes
2 answers
62 views

How are hidden layers counted / semantically defined?

I'm working my way through how LLMs work and I understand how things work but it's not clear to me exactly what is semantically defined as a "layer". Using the following FFN as an example: ...
0 votes
1 answer
30 views

Do Quo et al (2013) perform backpropagation between layers?

Le et al. 2013's non-weight sharing CNN has inspired me to ask two questions on this site previously. When training the three-layer autoencoder, do they compute dL/dW (where L is equation 1) ...
0 votes
1 answer
57 views

Is it possible training accuracy never changed while training?

Question summary What informations can get from this epoch_accuracy graph? Is it possible training accuracy never changed like after 10 epoch in graph while training? Body I do some experiments with ...
0 votes
0 answers
24 views

Permute inputs of neural network and expect different outputs?

My question is about if it makes sense to define a function (MLP) that takes two feature vectors f1 and f2. However, I want MLP(f1, f2) != MLP(f2, f1). I believe ...
0 votes
0 answers
47 views

Why does my loss function fluctuate so much?

I have a loss function that I'm trying to maximise using a neural network. While it does appear to increase and plateau over the training, it does so in a very "noisy" manner, spiking up and ...
1 vote
1 answer
88 views

Why is training all layers at a time effective for a multi-layer autoencoder?

This training of all layers of a CNN simultaneously is standard practice today. It is found in every CNN (AlexNet (2012), VGG, Inception, GANs, etc) and even pre-CNN networks such as Le et al. 2012. ...
0 votes
1 answer
26 views

Can a concept/feature be represented using more than one layer of a Neural Network?

I was reading Goodfellow. At the start of the text it was mentioned that there are two ways to represent depth of a deep neural network. One is using the depth of the computation graph and the other ...
2 votes
1 answer
232 views

Calculating mutual information between layer outputs and targets in a neural network

I've seen in several papers that it is possible to calculate the mutual information between a layer's outputs and the desired outputs. For example: Source: https://www.ncbi.nlm.nih.gov/pmc/articles/...
17 votes
4 answers
6k views

Did Minsky and Papert know that multi-layer perceptrons could solve XOR?

In their famous book entitled Perceptrons: An Introduction to Computational Geometry, Minsky and Papert show that a perceptron can't solve the XOR problem. This contributed to the first AI winter, ...
0 votes
0 answers
83 views

How to generate original training videos based on existing videoset?

I am a software engineer who is quickly ramping up on AI tech, but am nevertheless very new to the sector. A collegue has an extensive collection of training videos, the vertical is wheelchair seating ...
3 votes
1 answer
378 views

Are transformer models better than comparable-complexity MLP-based models?

I've watched the outstanding Andrej Karpathy's From Zero to Hero course. In the last lecture, he introduces Transformer decoder architecture, which is able to produce Shakespear-like text. However, ...
1 vote
0 answers
138 views

How can an MLP be implemented with convolutional layers?

I am studying the architecture of the network pointnet, specifically the MLPs stages of the pipeline highlighted in red in the following image (taken from the author page here): It is strange to find ...
19 votes
1 answer
974 views

What is the number of neurons required to approximate a polynomial of degree n?

I learned about the universal approximation theorem from this guide. It states that a network even with a single hidden layer can approximate any function within some bound, given a sufficient number ...
4 votes
1 answer
113 views

Backpropagation equation for a variant on the usual Linear Neuron architecture

Recently I encountered a variant on the normal linear neural layer architecture: Instead of $Z = XW + B$, we now have $Z = (X-A)W + B$. So we have a 'pre-bias' $A$ that affects the activation of the ...
3 votes
0 answers
55 views

Why does a neural network struggle to solve this simple problem?

Consider the following problem: Given a vector x of size dim with values between 0 and 1 (exclusive), determine if ...
0 votes
0 answers
46 views

Are autoencoders computationally cheaper than MLPs with the same number of neurons?

Are autoencoders computationally cheaper than other neural networks such as MLP with the same number of neurons? I have read in some papers that autoencoders train the network faster, and I could ...
2 votes
2 answers
145 views

Why is a simple regression problem so hard for an MLP to learn?

Consider a very simple problem, which is to find the maximum value out of a list of 5 numbers between 0 and 1. This is obviously trivial, but serves as a good example for a real-world problem I'm ...
1 vote
0 answers
63 views

Model unfit for some part of spiral data despite low error

I'm current testing a model for spiral data. After 500 epoches, loss is 0.04 but the result is still unmatch with some part of the training data. (bottom left) (source: upsieutoc.com) The model has 2 ...
3 votes
1 answer
127 views

Why is automatic differentiation still used, if today's computers can calculate symbolic derivatives quite fast?

Today's computers can calculate symbolic derivatives quite fast, why is automatic differentiation still used? For example, Mathematica can handle algebraic operations with arrays. Doesn't automatic ...
3 votes
3 answers
860 views

How can a neural network learn to play sudoku?

I'm just beginning to understand neural networks and I've performed a couple of successful tests with numerical series where the NN was trained to find the odd one or a missing value. It all works ...
1 vote
2 answers
61 views

What are all the possible usages of 'multilayer perceptron'?

The term 'multilayer perceptron' has been used in literature in various ways in the literature. I am presenting some of them below As a feed-forward neural network [1]. As a fully connected feed-...
2 votes
1 answer
100 views

Recent algorithms for correcting mislabeled data using multilayer perceptrons

I am doing literature research on algorithms for correcting mislabeled data using multilayer perceptrons. Found an "old" paper An algorithm for correcting mislabeled data (2001) by Xinchuan Zeng et al....
5 votes
1 answer
325 views

How to deal with padded inputs in a fully connected feed forward network?

I have a fully connected network that takes in a variable-length input padded with 0. However, the network doesn't seem to be learning and I am guessing that the high number of zeros in the input ...
0 votes
2 answers
255 views

Concise and mathematically-oriented book on AI and neural networks suitable as a gift [closed]

I would like to buy a book about AI and neural networks written on accessible level for a 17 years old mathematically very gifted student interested in these topics. The book should contain some ...
5 votes
2 answers
1k views

Why is it called back-propagation?

While looking at the mathematics of the back-propagation algorithm for a multi-layer perceptron, I noticed that in order to find the partial derivative of the cost function with respect to a weight (...
0 votes
1 answer
393 views

How to make a proper approximation of Sine function with Neural Networks?

TL;DR; How to build a neural network that properly approximates the sine function with different ranges? Context and Question: From this question I decided to use the Sergey's answer, however I used a ...
1 vote
1 answer
83 views

Does a second-order fully-connected layer have any uses?

I was thinking about implementing second-order regression via a fully-connected layer, and I came up with this: $X$ is the input data, shaped $(features, batch\_number)$. $w0$ is the bias, shaped $(...
1 vote
1 answer
83 views

Can RNNs get inputs and produce outputs similar to the inputs and outputs of FFNNs?

RNN and LSTM models have many architectures that can be modified. We can also compose their input and output data. However, in the examples that I found on the web, the inputs and outputs of RNNs/...
3 votes
3 answers
2k views

Why must the momentum factor be in the range 0-1?

Why is it a bad idea to have a momentum factor greater than 1? What are the mathematical motivations/reasons?
1 vote
0 answers
72 views

Is the capability of RNN more than the capability of MLP?

Consider the following excerpt paragraph taken from the section titled "Recurrent Neural Networks" of the chapter 10: Sequence Modeling: Recurrent and Recursive Nets of the textbook named ...
1 vote
0 answers
72 views

Does Godel's incompleteness theorems restricts the scope of connectionist-AI?

It is well-known that Godel's incompleteness theorems restricted the reachability of symbolic-AI, which is dependent on mathematical logic. But, I am wondering whether it has any impact on the ...
1 vote
1 answer
180 views

Are the capabilities of connectionist AI and symbolic AI the same?

The universal approximation theorem says that MLP with a single hidden layer and enough number of neurons can able to approximate any bounded continuous function. You can validate it from the ...
2 votes
1 answer
415 views

Why is the backpropagation algorithm used to train the multilayer perceptron?

I've read in the book Neural Network Design, by Martin Hagan et al. (chapter 11), that, to train the feed-forward neural network (aka multilayer perceptron), one uses the backpropagation algorithm. ...
1 vote
1 answer
822 views

What is the minimum number of neurons and hidden layers needed to learn a Boolean function that maps $N$ bits to $1$ bit?

Suppose I have a Boolean function that maps $N$ bits to $1$ bit. If I understand correctly, this function will have $2^{2^N}$ possible configurations of its truth table. What is the minimum number of ...
1 vote
0 answers
69 views

Is the VC dimension of a MLP regressor a valid upper bound on how many points it can exactly fit?

I want to calculate an upper bound on how many training points an MLP regressor can fit with ~0 error. I don't care about the test error, I want to overfit as much as possible the (few) training ...
6 votes
2 answers
2k views

Can neurons in MLP and filters in CNN be compared?

I know they are not the same in working, but an input layer sends the input to $n$ neurons with a set of weights, based on these weights and the activation layer, it produces an output that can be fed ...
1 vote
1 answer
276 views

When should we use CNN instead of MLP?

Is CNN only applicable to time-series data or image data? When should we use CNN instead of MLP?
0 votes
1 answer
2k views

What, exactly, do mlp(64,64) and mlp(64,128,1024) mean in PointNet, and how many input neurons does 1 (x,y,z) point have?

I couldn't find out how to interpret the multilayer perceptron notation given in PointNet. Specifically, I am looking to find out what the numbers inside the parentheses of ...
1 vote
1 answer
521 views

Why doesn't the LSTM model improve the time-series forecasting significantly with respect to the MLP model?

I have recently started learning time series forecasting. I have a dataset of the weekly payment history of 10k clients over 1 year, and I want to predict the future 5 payments for a test set of 1k ...
0 votes
1 answer
110 views

About the choice of the activation functions in the Multilayer Perceptron, and on what does this depends?

I've read in this: F. Rosenblatt, Principles of neurodynamics. perceptrons and the theory of brain mechanisms that in the Multilayer Perceptron the activation functions in the second, third, ..., are ...
0 votes
0 answers
68 views

What are the math theorems regarding the Multilayer Perceptron?

I've come across a theorem "Convergence theorem Simple Perceptron" for the first time, here-> https://zaguan.unizar.es/record/69205/files/TAZ-TFG-2018-148.pdf, page 27, (is in Spanish) ...