Tag Info

8

There does not appear to be a historical consensus on this. The Wikipedia page on the Perceptrons book (which does not come down on either side) gives an argument that the ability of MLPs to compute any Boolean function was widely known at the time (at the very least to McCulloch and Pitts). However, this page gives an account by someone present at the MIT ...

7

An RNN or LSTM have the advantage of "remembering" the past inputs, to improve performance over prediction of a time-series data. If you use a neural network over like the past 500 characters, this may work but the network just treat the data as a bunch of data without any specific indication of time. The network can learn the time representation only ...

6

Why is it called back-propagation? I don't think there is anything special here! It's called back-propagation (BP) because, after the forward pass, you compute the partial derivative of the loss function with respect to the parameters of the network, which, in the usual diagrams of a neural network, are placed before the output of the network (i.e. to the ...

6

You are talking about two different types of 'size'. The size of the input for a FFNN and a RNN must always remain fixed for the same network architecture, i.e. they take in a vector $x \in \mathbb{R}^d$ and could not take as input for instance a vector $y \in \mathbb{R}^b$ where $b \neq d$. The size you refer to in the context of the RNN is the length of ...

5

Inherently, no. The MLP is just a data structure. It represents a function, but a standard MLP is just representing an input-output mapping, and there's no recursive structure to it. On the other hand, possibly your source is referring to the common algorithms that operate over MLPs, specifically forward propagation for prediction and back propagation for ...

4

There are a ton of sample datasets our there you can play with. A bunch of good ones install with R in the datasets package. Luckily you can download them independently if you're not an R user. Try https://vincentarelbundock.github.io/Rdatasets/datasets.html You might also be interested in the MNIST database which is one of the canonical databases used ...

4

According to wikipedia of backpropagation: In fitting a neural network, backpropagation computes the gradient of the loss function during supervised learning with respect to the weights of the network for a single input–output example, and does so efficiently, unlike a naive direct computation of the gradient with respect to each weight individually. ...

3

Assumptions Different model structures encode different assumptions - while we often make simplifying assumptions that aren't strictly correct, some assumptions are more wrong than others. For example, your proposed structure of "just pass the $X$ number of letters leading up to the last letter into an FFNN" makes an assumption that all the information ...

3

Sure, you can define plenty of things we don't generally need to regard as recursive as so. An MLP is just a series of functions applied to its input. This can be loosely formulated as $$o_n = f(o_{n-1})$$ Where $o_n$ is the output of layer $n$. But this clearly doesn't reveal, much does it?

3

tl;dr The equivalent to a neuron in a Fully-Connected (FC) layer is the kernel (or filter) of a Convolution layer Differences The neurons of these two types of layers have two key differences. These are that the convolution layers implement: Sparse connectivity, i.e. each neuron is connected only to an area of the input, not the whole. Weight sharing, i.e. ...

3

In neural networks, the family of functions and the shapes that they can make for decision surfaces is determined by the activation function you use (in your case, tanh or hyperbolic tangent). Assuming at least one hidden layer, then the universal approximation theorem applies. How closely you can approximate any given function is limited by the number of ...

3

Well, adding gaussian noise is a very common regularisation method. Maybe this paper is interesting to you. They also have very small datasets. In the end there is only so much you can get out of a given dataset.

3

A MLP only does pattern recognition, it will not learn search. Tictactoe, (Oughts and Crosses), is such a simple game that your network should learn the moves from the training data by heart, no generalisation required. If it still loses games, maybe your training data doesn't consist of particularly good moves.

3

As you stated, it's popular to have some form of a rectified linear unit (ReLU) activation in hidden layers and the output layer is often a softmax or sigmoid (depending also on the problem: multi-class or binary classification, respectively), which provides an output that can be viewed as a probability distribution. You could generalize this further to ...

3

Yes, there are different ways. What I think you are looking for is under the research field of Localization and Mapping. Which divides in the following subfields: For getting current (the robot) position and trajectory go to models for Odometry Estimation For getting a representation of the world around the robot go to models for Mapping If you want both of ...

3

CNN applies the same filters to every "chunk" of the input data. It's applicable when you think every chunk should be processed the same way. For example, we think a face in the top-left of the image should be recognized just as well as a face in the bottom-right. So we do expect to process each part of the image the same way, and a CNN is good for ...

2

A popular dataset is the fisher iris dataset. It consists of 150 samples each with a dimensionality of 4. You can find it at http://archive.ics.uci.edu/ml/datasets/Iris

2

Whether Minsky knew or not, it was definitely known to Rosenblatt, as he published those results in his really pioneering report - Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms, published in 1961. A large majority of academic and industry experts are simply unaware of the "depth" of Rosenblatt's publication on ...

2

In section 13.2 Other Multilayer Machines (pp. 231-232) of the book Perceptrons: An Introduction to Computational Geometry (expanded edition, third printing, 1988) Minsky and Papert actually talk about their knowledge of or opinions about the capabilities of what they call the multilayered machines (i.e. perceptrons with many layers or MLPs). Have you ...

2

The short answer to your question is: you probably do not fully know your data. remember that ML is no magic wand. It needs your understanding of the data and the behavior of it. Although it is approved that neural networks with at most two hidden layers can approximate any model with an acceptable degree precision, setting up the structure of the neural ...

2

You can take a look at this paper that solving your problem with a neural network. You can use the pytorch implementation of the satnet layer : satnet layer API. In this supervised setup the layer also learn the boolean constraints of your model. You can find an example of a sodoku solver in the github repo.

2

I think it is the wrong way to frame sudoku as a regression problem in neural networks. Firstly, you have to understand what regression is. "Regression" is when you predict a value given certain parameters, where the parameters are related to the value you have to predict. This happens because at the core neural networks are "function approximators", they ...

2

I think that making some draws might help. Below I tried to draw the model architecture. We start with classic feed-forward structure: input represented by a vector I with length f (number of features), a hidden layer H which does not have a fixed size, and output O of length c (number of classes). Then we have 3 extra vectors than usual: a vector U they ...

2

You probably got the back propagation wrong. I have done a test on the accuracy on adding an extra layer and the accuracy went up from 94% to 96% for me. See this for details: https://colab.research.google.com/drive/17kAJ2KJ36grG9sz-KW10fZCQW9i2Tf2c To run the notebook click Open in playground and run the code. There is a commented line which add 1 extra ...

1

You can use MNIST obviously but I'd also suggest you have a look at UC Irvine's datasets: https://archive.ics.uci.edu/ml/datasets.php

1

Have a look at the following article Principles of training multi-layer neural network using backpropagation. It was very useful to me. You can also see here an example of backpropagation in Matlab. It effectively solves the XOR problem. You can also play around with the cost function or the learning rate. You may get surprising results! Does this answer ...

1

Simply said, predicting pseudo random number is just not possible for now. Pseudo random numbers generated now have a high enough "randomness" so that it cannot be predicted. Pseudo random numbers is the basis of modern cryptography which is widely used in the world wide web and more. It may be possible in the future through faster computers and stronger AI, ...

1

A common model used for this kind of classification task is to have one output neuron per class. So, for example, neuron 1 may have a loss function that is related to outputting "1" for examples of class 1, and "0" for examples of other classes. Neuron 2 may be asked to do the same, but for class 2 rather than class 1. If you use a model of this kind, you ...

1

This depends on whether the output is a continuous or discrete variable. If the output variable is discrete (there are a finite number of possibilities that it can be), as in a classification task (such as this one, where you are trying to place the input into one of 5 categories), you want to use one output neuron for each class. If the variable is ...

1

The forward prop equation is: $$Z = (X-A)W - B = XW - AW - B$$ So the derivatives for $Z$ w.r.t $W$, $A$, $B$ and $X$ should be: $$\frac{\partial Z}{\partial W} = X-A \\ \frac{\partial Z}{\partial A} = - W \\ \frac{\partial Z}{\partial B} = - 1 \\ \frac{\partial Z}{\partial X} = W$$ I don't know why he needs the last one though. The first is, like ...

Only top voted, non community-wiki answers of a minimum length are eligible