I'm completely new to AI and admittedly have never been good at math (also please excuse me if I use the wrong terminology). Despite this, I'm trying wrap my head around activation functions and how they are used by a neural network to fit a line to a set of data.
I understand the purpose of an activation function, but what I'm struggling with is how we go from deciding which type of activation function to use for each neuron to actually fitting a line to our data.
In an attempt to illustrate this question, here is a YouTube video which demonstrates how a fitted line would change as data is passed through a neural network using various types of activation functions.
At the moment, it just seems like magic that we pass data through and depending on which type of activation function being used the line somehow changes in various ways and I have no idea why it's doing what it's doing.
Any help with this or guidance regarding what knowledge gaps I might have that are preventing me from understanding this would be greatly appreciated! My goal in all in asking this is to be able to do the math by hand in a much smaller network to demonstrate that I understand what's happening behind the scenes.