2
$\begingroup$

I am new to Neural Networks and my questions are still very basic. I know that most of neural networks allow and even ask user to chose hyper-parameters like:

  • amount of hidden layers
  • amount of neurons in each layer
  • amount of inputs and outputs
  • batches and epochs steps and some stuff related to back-propagation and gradient descent

But as I keep reading and youtubing, I understand that there are another important "mini-parameters" such as:

  • activation functions type

  • activation functions fine-tuning (for example shift and slope of sigmoid) types of sigmoid finetuning

  • whether there is an activation funciton in the output

  • range of weights (are they from zero to one or from -1 to 1 or -100 to +100 or any other range)

  • are the weights normally distributed or they just random

etc...

Actually the question is:

Part a:

Do I understand right that most of neural networks do not allow to change those "mini-parameters", as long as you are using "readymade" solutions? In other words if I want to have an access to those "mini-parameters" I need to program the whole neural network by myself or there are "semi-finished products"

Part b:(edited) For someone who uses neural network as an everyday routine tool to solve problems(Like data scientist), How common and how often do those people deal with fine tuning things which I refer to as "mini-parameters"? Or those parameters are usually adjusted by a neural network developers who create the frameworks like pytorch, tensorflow etc?

Thank you very much

Improve this question
$\endgroup$
5
  • $\begingroup$ You can look at the API of whichever library you're using to see which hyperparameters you can set yourself, e.g. scikit-learn lists a bunch. The default values are decent though. $\endgroup$
    – NotThatGuy
    Oct 11, 2020 at 1:49
  • $\begingroup$ Often you'd need to adjust the weights yourself so it's in the expected range (which is called "normalisation"). The distribution of values may affect the result and there may hyperparameters to indirectly deal with an improper distribution (like providing weights for different classes), but I wouldn't expect there to be an parameter for specifying the distribution directly. I don't think it would be able to do too much with that information. $\endgroup$
    – NotThatGuy
    Oct 11, 2020 at 1:54
  • $\begingroup$ NotThatGuy , thanks. Can you please make me some order in understanding. Is it right to say: that things like pytorch, tensorflow, keras, scikit learn are alternatives one to another? Thay all allow to build me a desired network with all hyperparameters ibe mentioned? Are they all called "frameworks"? $\endgroup$
    – Igor
    Oct 11, 2020 at 8:11
  • $\begingroup$ @Igor I'm unfortunately not familiar enough with the others to be able to answer that, but you can certainly create neural networks using scikit-learn (with whichever hyperparameters it supports). Wikipedia calls that a "library". I'll leave it to other people to explain the difference between a framework and a library and judge what is a library and what is a framework. $\endgroup$
    – NotThatGuy
    Oct 12, 2020 at 15:48
  • $\begingroup$ NotThatGuy , thanks again. You got my question right. BTW It is not that I am trying to learn all frameworks at once, I just understand things better when I have some kind of taxonomy (tree like structure)of relevant terms inside my head. Like which term is a private case of which. Which term is a generalization of some other terms, and which terms are equal in hierarchy (they may be not totally equal but more alternatives) $\endgroup$
    – Igor
    Oct 13, 2020 at 7:52

1 Answer 1

Reset to default
3
$\begingroup$

In general, many of the parameters you mentioned are called hyperparameters. All hyperparameters are user-adjusted (or user-programmed) in training phase. Some hyperparameters are:

  • learning rate,
  • batch size,
  • epochs,
  • optimizer,
  • layers,
  • activation functions etc.

To answer your (a) part of your question, there are obsiously many frameworks and libraries, for example in python; TensorFlow, pytorch and so on. You might never create a net from the very beginning; maybe only in order to understand the forward and backpropagation algorithms. When we call from scatch networks, we mean that these networks are trained from scratch, with learnable weights and chosen hyperparameters; with no transfer learning.

To answer your (b) part of your question, I can understand from it that you mean when a net is good enough. Dependently of your data, of course, a neural network is good enough, when it is trained adequately on them. That is, you should be aware of overfitting, underfitting, and in general of the model you are trying to train with all its parameters and hyperparameters.

Since, you are at the very beginning with Machine Learning, I propose you read some books, in order to get everything needed, in terms of Mathematical and Computer Science aspects.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Hi thanks for answer. About part a: Do I get right that "FRAMEWORKS" like those you had mentioned (tensor-flow and pytorch) are NOT ready architectures but kind of a sandbox which allows me to build my own? including the adjusting all hyperparameters like those I've mentioned above? (weights range, weights distribution, option to decide in layers to use or not use activation functions, and if use which kiind of them)... Do I get right? About part b: is not what mean. I'll edit this part of a question. $\endgroup$
    – Igor
    Oct 11, 2020 at 7:49
  • 1
    $\begingroup$ @Igor as ddaedalus mentioned above, "You might never create a net from the very beginning; maybe only in order to understand the forward and backpropagation algorithms". I am sure with you with the involution of frameworks in this machine learning era, nothing you can't do with them. You can create a new model, tune all parameters, all weights range, change other SOTA model, and they also open the source code of those frameworks, so if you have any problems, feel free to change it, or contribute it to the community. $\endgroup$
    – CuCaRot
    Oct 11, 2020 at 8:49
  • 1
    $\begingroup$ There are some differences between those frameworks but it doesn't affect you right now, just choose which has syntax you feel comfortable with and try to read and understand as much as possible. Welcome to this world! $\endgroup$
    – CuCaRot
    Oct 11, 2020 at 8:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .