Looking for an explanation of the linear regression estimation method in deep learning.
There's no point to fit a linear regression model (such as OLS) with neural network because it's really designed for non-linear models. But if you want to do that, you'll just need to set linear activation units.
Generally speaking, you can say this:
there is a relationship between neural network learning (I'm assuming a "vanilla" ANN here, no CNN's or RNN's or anything) and linear/logistic regression.
But they're not the same thing. Just related. You could maybe consider them "cousins" to use a real-life analogy.
The big obvious difference is this: standard linear regression is, well, linear, that is, it's based on a straight line. So it can only separate points on a plane which can be separated by a straight line drawn on that plane. An ANN however, is non-linear and can fit all sorts of crazy looking curves. The reason why this is true has to do with a combination of the "activation" functions that are used, as well as the layering effect of your hidden layers.
To be fair, if you extend linear regression to be polynomial regression, you can fit more complicated curves, but that has its own downsides. And while they are also related, linear regression and polynomial regression aren't - strictly speaking - the same thing (although they may both be special cases of the same general technique).
All of that may be over-simplifying a bit. If you really want a good explanation of both linear/logistic regression and ANN's and some explanation of how they relate and differ, I recommend Andrew Ng's ML courses on Coursera. Both the original one and the new DeepLearning.ai ones.