7

$\max(-y_i(w x_i), 0)$ is not partial derivable respect $w$ if $w x_i=0$. Loss functions are problematic when not derivable in some point, but even more when they are flat (constant) in some interval of the weights. Assume $y_i = 1$ and $w x_i < 0$ (that is, an error of type "false negative"). In this case, function $[y_i - \text{sign}(w x_i)]^2 ...


7

There does not appear to be a historical consensus on this. The Wikipedia page on the Perceptrons book (which does not come down on either side) gives an argument that the ability of MLPs to compute any Boolean function was widely known at the time (at the very least to McCulloch and Pitts). However, this page gives an account by someone present at the MIT ...


4

It can be done. The activation function of a neuron does not have to be monotonic. The activation that Rahul suggested can be implemented via a continuously differentiable function, for example $ f(s) = exp(-k(1-s)^2) $ which has a nice derivative $f'(s) = 2k~(1-s)f(s)$. Here, $s=w_0~x_0+w_1~x_1$. Therefore, standard gradient-based learning algorithms are ...


4

I assume you're talking about a perceptron threshold function. One definition of it with an explicit threshold is $$f(\textbf{x})= \begin{cases} 1& \text{if } \textbf{w}\cdot\textbf{x} > t\\ 0& \text{otherwise} \end{cases}.$$ Another form with a bias is $$f(\textbf{x})= \begin{cases} 1& \text{if } \textbf{w}\cdot\textbf{x} + b > 0\\ 0&...


3

Since we're dealing with real-values variables, it is almost certainly the case that the argument of the function will not be $0$. If you care strongly about that point, you can just use sub-gradients instead (and we do have sub-gradients for this function, so there is no problem).


2

It is important to note that the exact statement is the eqation given below can never be 0 for misclassified points in $ S^+$ $$ E(X) = (y - \text{sign}\{\overline{W} \cdot \overline{X}\}) $$ And $S+$ is defined as the set of all misclassified training points $X \in S$ that satisfy the condition $y(\overline{W} \cdot \overline{X})<0 $ which means that $y$ ...


2

Y is the desired output of the perceptron (often referred to as target) , for the given set of input vectors. Rationale behind Y.a<=0 : Prerequisite knowledge : A=A-B : Moves vector A away from direction of vector B A=A+B : Moves A in the direction of B A (.) B >0 ; A vector is directed acutely (<90 deg.) towards B vector A (.) B <0 ; A vector is ...


2

In Brief: re-train your dataset. I believe where you get lower accuracy scores, your model has not converged to the final state. duplicate your dataset multiple times and create a bigger one, then train your model with it. In Detail: number_of_samples_classified_correctly/total_number_of_samples(I'm not sure this should be the correct definition for ...


2

The main problems are that your activation function is not monotonic (as pointed out by csrev), and that it is not continuously differentiable. These make it very difficult / impossible to use standard gradient-based learning algorithms. So yes, there may exist a good solution of weight values, but it is very difficult to find or approximate those weight ...


2

Indeed I think the problem is with the way you've defined the activation function. By selecting it arbitrarily, you could solve many specific problems. In practice, activation functions used are monotonic. It keeps the error function convex at a per-layer level. In theory though I'm not sure exactly what Rosenblatt has claimed so it might be worth calling ...


2

I think your confusion comes from the fact that you are calling those two hidden nodes "perceptrons". You shouldn't call the hidden nodes in your network perceptrons. You should call them "nodes" or "neurons", although the term "multilayer perceptron" comes exactly from that. You should think of a perceptron as something like A perceptron simply computes a ...


2

The section of the book Perceptrons: An Introduction to Computational Geometry (expanded edition, third printing, 1988) that shows the limitations of the perceptron should be 11.8 The Nonseparable Case (p. 181), where the authors write There are many reasons for studying the operation of the perceptron learning program when there is no $\mathbf{A}^*$ with ...


1

I will first address your main question "Why did the development of neural networks stop between 50s and 80s?" In 40-50s there was a lot of progress (McCulloch and Pitts); the perceptron was invented (Rosenblatt). That gave rise to an AI hype giving many promises (exactly like today)! However, Minsky and Papert have proved in 1969 that a single-...


1

I will tell you my knowledge, correct me if I am wrong. Perceptron Learning Algorithm (PLA) is a simple method to solve the binary classification problem. Define a function: $$ f_w(x) = w^Tx + b $$ where $x \in \mathbb{R}^n$ is an input vector that contains data points and $w$ is a vector with the same dimension as $x$ which present for the parameters of our ...


1

Before proving that XOR cannot be linearly separable, we first need to prove a lemma: Lemma 1 Lemma: If 3 points are collinear and the middle point has a different label than the other two, then these 3 points cannot be linearly separable. Proof: Let us label the points as point $A$, $B$, and $C$. $A$ and $C$ have the same label, and $B$ has a different ...


1

In Single Perceptron / Multi-layer Perceptron(MLP), we only have linear separability because they are composed of input and output layers(some hidden layers in MLP) This is wrong. A multi-layer perceptron (i.e. a feed-forward neural network) with non-linear activation functions can perform non-linear classification and regression. In fact, an MLP with one ...


1

This is one of the main skills that separates someone with a deep understanding of, and experience in, machine learning learning, from a neophyte. There are several approaches: Try several methods, perhaps with automated hyperparameter optimization, and see if there's a big difference in typical model quality. This is pretty common if you don't have a lot ...


1

It seems I've solved the issue. There was several mistakes: 1. I've generated random weights from 0 to 1. As a result, too big numbers passed through softmax function (>10000), and the function wasn't calculated correctly. I divided each initial weight on the number of neurons in previous layer and solved the issue. 2. I've calculated separate delta for ...


1

The paper (or report) that formally introduced the perceptron is The Perceptron — A Perceiving and Recognizing Automaton (1957) by Frank Rosenblatt. If you read the first page of this paper, you can immediately understand that's the case. In particular, at some point (page 2, which corresponds to page 5 of the pdf), he writes Recent theoretical studies by ...


1

This is a very dicey question. Logic functions can be thought of as mapping multiple inputs to a single output. Now each logic function create its own boundary. So if you are using a complex logical equation it is actually very hard to approximate the underlying function. Here I am treating the input Booleans as the input features. From practical experience:...


1

A perceptron is a linear threshold function. That means it has a weight vector $w$, and it outputs $w \cdot x > t$, where $x$ is the input vector and $t$ the threshold. Naïve Bayes makes the assumption that all features are independent (hence the term naïve). It predicts the most likely class by using Bayesian probability, for each class multiplying the ...


1

Naive Bayes is a generative algorithm while Perceptron is a discriminative algorithm. That is the main difference.


Only top voted, non community-wiki answers of a minimum length are eligible