3

A neural network can be reduced to a linear regression model only if we use linear activation functions (i.e. $\sigma(x) = x$), and only if we do not use any neural network specific techniques such as convolution, residuals, etc., as shown below: $\text{neural network}(x) = \sigma_n(W_{n} \sigma_{n-1}(W_{n-1}\dots\sigma_1(W_1 x + b_1) + \dots + b_{n-1}) + ...


3

In some sense, you're right that a neural net is just another tool to fit data. However, it's quite the tool! There's this universal approximation theorem saying that, under decent conditions, a neural network can get as close as you want to a wide class of functions. This means that you can get the network to give you complicated shapes with squiggles all ...


3

The fact that features are always positive values don't guarantee that outputs of hidden layers are positive too. Due to multiplication, output of an hidden layer could contain negative values, i.e., a hidden layer can contain weights that have opposites signs as its input. Remember that only layer outputs, not their weights, are passed through ReLu, so, ...


2

Softmax is a probability distribution you use when you want probability for all multiple classes you are predicting which are not independent, ie, exp(xi)/sum(exp(xj) for j in all x), where xi is the score of one neuron, so softmax is good if you have more than 1 neurons, but for just 1 neuron(in this case), the output of softmax will be 1, always (exp(xi)/...


2

The difference is simply that non-linear regression learns parameters that in some way control the non-linearity - e.g. any weight or bias that is applied before a non-linear function. For instance: $$y = (w_1 x_1 + w_2 x_2)^2 + w_3$$ With such a function to learn, you cannot separate out transformed values of $w_1$ and $w_2$ and turn this into a linear ...


2

There's no point to fit a linear regression model (such as OLS) with neural network because it's really designed for non-linear models. But if you want to do that, you'll just need to set linear activation units.


1

neural networks can solve all taylor series polynomials meaning a NN is an generalized linear model. Most function f(y) can be solved with neural networks. However, many matrix operations can not be generalized for a neural network to solve like determinants. Operations like rotation, scale, and transform also can not be generalized. you can solve all ...


1

Generally speaking, you can say this: there is a relationship between neural network learning (I'm assuming a "vanilla" ANN here, no CNN's or RNN's or anything) and linear/logistic regression. But they're not the same thing. Just related. You could maybe consider them "cousins" to use a real-life analogy. The big obvious difference is this: standard ...


Only top voted, non community-wiki answers of a minimum length are eligible