5
$\begingroup$

I've been studying geometry and linear algebra for months with the goal to build neural networks. But now I'm reading that perceptrons require fitting curves, and curves are not expressed as linear functions. So, I might need to study differential geometry and calculus for building good fitting curves in perceptrons.

I already know how to code and was hoping to get my hands dirty by coding a few neural networks. But should I study calculus and differential geometry before coding?

From this video, I understand that the least squares approximation can be used to fit a curve through a set of points, so maybe linear algebra is enough for building good neural networks?

Improve this question
$\endgroup$
4
  • 6
    $\begingroup$ What level are you hoping to build neural networks at? You only need a brief overview of calculus and statistics if you want to use a toolkit like Keras or PyTorch. If you want to understand all the details, and build everything yourself in C++ or similar, then your need more. $\endgroup$
    – Neil Slater
    Aug 6, 2021 at 7:44
  • $\begingroup$ Building it from scratch. Not using any external library. I've already studied statistics and probability. Ok so I need Calculus. $\endgroup$
    – user456280
    Aug 6, 2021 at 8:01
  • 2
    $\begingroup$ As a side note, "perceptrons" and "neural networks" may not be the same thing. People usually use the term perceptron to refer to a very simple neural network that has no hidden layer. Maybe you meant the term "multi-layer perceptron" (MLP). $\endgroup$
    – nbro
    Aug 6, 2021 at 10:17
  • 2
    $\begingroup$ @nbro FYI: I asked a new question based on your comment. $\endgroup$
    – R.M.
    Aug 7, 2021 at 13:48

2 Answers 2

Reset to default
13
$\begingroup$

Neural networks are essentially just repeated matrix multiplications and applications of an activation function, so you really don't need a great deal of linear algebra to construct a simple neural network — if you understand how to multiply matrices, that's probably sufficient.

The harder bit is the training process which is typically done through backpropagation. You need a bit of calculus, but differential geometry is overkill. There are some interesting topics in differential geometry for machine learning, but it's far beyond what is needed to implement backpropagation.

To understand backpropagation you just need to know about the gradient of a function and what this means intuitively; you should also have a good knowledge of the chain rule. That's really all you need, and any course on "multivariable calculus" or something similar would give you more than enough to get started.

Of course, it never hurts to know more, but fortunately neural networks are simple enough that you don't need to struggle for years before you can implement a basic neural network; try to get started as soon as you have the basics, and learn the rest as you go.

Improve this answer
$\endgroup$
0
1
$\begingroup$

To give some practical advice, it is important to understand parts of calculus. This is mainly because Backpropagation is a leaky abstraction in modern libraries. In a nutshell, there is a lot which can go wrong (exploding or vanishing gradient for example) and you will need knowledge about gradient descent to handle it.

I highly recommend Andrej Karpathys Lecture on it. He gives a easy to understand and intuitive explanation.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .