1
$\begingroup$

I have become more familiar with libraries such as tensorflow for a while now, and have become interested in utilizing neural networks for solving specific problems. The big question I have is, what are some principles that you have to take into account for designing your neural networks architecture?

Some other questions I have are:

  • Do I want my network to slowly reduce the dimensionality of data (so it picks out important features), the deeper it goes? What happens when the output (lets say its one hot encoded, so the no. classes is in the 1000s while your e.g text is only of length 30) is a lot bigger than the input?
  • If so, then what do I do when I have to process a single class? Do I just add layers which expand off that 1 input (isn't that wasting resources?)

What resources do you recommend I should look into?

$\endgroup$
3
  • $\begingroup$ I have provided a pretty lengthy answer, but ideally you would post a single question at a time. Additionally, your third question (second "other" question) is unclear. What do you mean with single class? A single output value? $\endgroup$
    – Kroshtan
    Jul 14, 2022 at 13:14
  • $\begingroup$ @Kroshtan, sorry about that, I just did not want the question to be flagged as vague, by providing some specific questions to answer. What I meant is that there is a single input value, e.g username_id. $\endgroup$ Jul 15, 2022 at 1:10
  • $\begingroup$ A sample with a single value input and a single value output is a lookup table, there is nothing to learn regardless of expanding dimensions. If the input and output is numeric, you could technically learn a pattern such as y=x^2, but that's it. $\endgroup$
    – Kroshtan
    Jul 15, 2022 at 16:28

2 Answers 2

2
$\begingroup$

Designing neural network architectures from scratch for harder tasks is work usually performed by entire research groups (whether academic or business). There are, however, some things to keep in mind:

  • Deeper networks have more abstraction, but also higher complexity. This means they can learn more complicated relations between input and output, but can also maximally encode the noise in the dataset. You may know this as the bias/variance trade-off, but is more often called over-fitting.

  • The type of data that goes into a network and the intuitive aspects of the data help decide what type of layers to use. Linear layers are your baseline. Convolutional layers when location is important (whether 1D, 2D or 3D). Attention layers for context, such as in language models. Recurrent connections for time-series data, such as speech patterns or observations over time.

  • Determining best practices on your own is an impossible task. There is a reason research is based on previous work. You may gain intuition over time, but decisions like when to use additional operations such as pooling, batch normalization, or which hyper parameters to use, such as activation functions, optimizers and learning rate, are best borrowed from existing work. Try to find work on a similar task to what you want to do and read some papers, preferably, or blogs and tutorials if the former is too complex.

If you have a specific task in mind, by all means post it as a comment to this answer and I can help you find similar work and/or suggest some intuitive starting point from my own experience.

In response to your second question: You do not need to reduce the dimensionality of your data from the first layers of your model. In fact, you can expand it considerably. In the end you need to boil it down to the format of your output. For example classification using a CNN will first create feature maps of your data, which will increase dimensionality (e.g. 256x256x3 image to 128x128x64 feature maps), then it will flatten the features (128x128x64 feature maps to 1048576 hidden nodes) and finally it will classify (1048576 hidden nodes to 1000 output classes).

$\endgroup$
1
$\begingroup$

I think this part is the art/experience part. The only certain thing is the deeper you go, the higher is the level of abstraction, and you don't always need your machine to philosophize.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .