In the image below taken from a Youtube video, the author explains that the neural network can be used to fit a relational graph for a set of data points shown by the green line. And that this is accomplished by using weights, biases and activation functions.

My slight confusion is that, initially, the weights and biases and randomized, and they are re-adjusted by backpropagation. This means that, at the end of the output layer, we must have the actual values of the target function anyway.

So what problem does the neural network really solve?

So, for example, we want to find the target function for dosage and efficacy, we are given the data points shown in blue. If we initially choose randomized values for the weights, biases and activation function, then, at the output layer, we determine an output value for efficacy, but there is no way to know whether this value is in fact correct or not. So, we need the actual values to determine the difference.

What about when we choose a value of dosage which has not been observed, for example, 0.25? Doesn't this rely upon a best-fit relation graph that has already been fitted to the data prior to adjusting the neural network?

enter image description here

  • $\begingroup$ Hello. Your questions are all legitimate, but, please, next time, ask only one question per post. If you have multiple questions, ask one for each post. $\endgroup$
    – nbro
    Sep 20 at 13:27
  • $\begingroup$ Note that real neural networks typically have lots of input dimensions. With only one input dimension, we can usually just interpolate the known data points. $\endgroup$
    – user253751
    Sep 20 at 21:30

This means that, at the end of the output layer, we must have the actual values of the target function anyway.

Yes, this is necessary for supervised learning. You will often see this called a labelled dataset, where the "label" is an output value that you know is associated with each input. A set of labels associated with some inputs, that you have collected for training may also the called the "ground truth".

We do not need all possible values though, but enough examples that the neural network can interpolate between them. How many examples that is depends on the complexity of the function we want to learn.

So what problem does the neural network really solve?

There are three main things it solves, and these are shared with most other machine learning approaches:

  • The neural network learns a function from examples of input and output.

  • The neural network will learn an expected value (or for classifiers, a probability distribution) when trained using noisy or stochastic data.

  • The neural network makes few assumptions about the relationship between input and output, and can learn successfully even for quite complex relationships.

These traits are all useful when you do not have a strong sense of what the correct function should be, in terms of writing an equation, but do have many examples of it.

What about when we choose a value of dosage which has not been observed, for example, 0.25?

The neural network will still produce an output. In a very simple scenario, where you had trained with example inputs at e.g. 0.2 and 0.3, then the output will likely be somewhere between the outputs for those two values. For a neural network, this in-between value can be much more sophisticated than a simple mean of the nearest examples. ML that uses the nearest values exists, that is called k-nearest neighbours.

If this process has worked well, and the trained neural network produces useful, accurate predictions from unseen inputs, it is said to generalise well. Very often, that is the goal for training a neural network or other machine learning system.

It is worth mentioning some additional facts and features of training neural networks (and ML in general):

  • When generalisation is the goal (and it often is), then you need to test for it. This is done by keeping some example data back, not using it to train, but instead using it to check results on unseen inputs. In fact, this is so important, the data is often split into three sets - a training set, a cross-validation (aka development) set, and a test set.

  • The lack of assumptions in the basic model can lead to needing many training examples. If you know something specific or useful about the function being learned, you can pre-process the inputs to help - this is called feature engineering.

  • Machine learning can be good at interpolation, i.e. calculating outputs for inputs that are not in the original training data, when the unseen inputs are in-between or close to the training examples. Even when good at interpolation, it will still be bad at extrapolating to new unseen inputs that are outside of the ranges of the training examples. That is because it has used a very general/flexible system to fit some line or curve to examples, it has not learned an analytical function.

Exceptions to all of these points exist. They are the norm, but it will depend on the details of what you are trying to do.

  • $\begingroup$ Thank you! appreciate the efforts you've put into your post! Will upvoted :) Do you have any recommendations for good books, articles? Ive made a note of the ones you provied alreayd $\endgroup$ Sep 18 at 17:47
  • $\begingroup$ I learned the basics from Coursera courses. coursera.org/learn/machine-learning is still very good because it doesn't dive into theory too heavily and teaches good practice (test/train split) from the beginning $\endgroup$ Sep 18 at 17:57
  • $\begingroup$ Okay ill take a look, thanks $\endgroup$ Sep 18 at 18:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.