1
$\begingroup$

You may not believe it, but I am an ANN expert. Perhaps, for that reason, I am unable to grasp completely what the layers are in a Deep Forward Artificial Neural Network (DFANN).

According to the Deep Learning "bible", p. 164 "the model is associated with a" directed acyclic graph (DAG) "describing how the functions are composed together". Then it states that one of these functions $f^{(i)}$ is a layer. We know that an ANN is composed of neurons/units (the biological perspective) and a DAG is composed of nodes/vertices and edges/links. A trivial mapping can be established between (neural) units and nodes, and (neural) connections and edges (see also Fig. 6.2 in p. 170). However, defining the layers in a DAG is not that trivial, I would say.

In a quick internet search, I came across about the concept of graph drawing and layering algorithms. This concept is never mentioned in the ANN literature. If I understood well, formally, we would need such an algorithm to be able to define the layers in the DAG.

A more intuitive description is given by AIMA: DFANN "are usually arranged in layers, such that each unit receives input only from units in the immediately preceding layer". That sounds indeed like a layering algorithm.

So, formally, what are the layers in an Artificial Neural Network?

As a side note, a practical argument is given by https://cs231n.github.io/neural-networks-1/

One of the primary reasons that Neural Networks are organized into layers is that this structure makes it very simple and efficient to evaluate Neural Networks using matrix vector operations.

That's nice, but only from a practical perspective.

An interesting remark has been made by Chillston (see comments, I quote):

That's because an FFN is not a general DAG but a very regular kind of DAG, right.

I also thought that could be the case. But Bishop, 1995, p. 116 contradicts us:

More generally we can consider arbitrary network diagrams (not necessarily having a simple layered structure) since any network diagram can be converted into its corresponding mapping function. The only restriction is that the diagram must be feed-forward, so that it contains no feedback loops.

As I see it, according to Bishop, any (general) DAG is indeed an ANN.

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ Can you specify your question or maybe describe what you mean exactly? From what I understand you are reasoning: An FFN is a DAG -> in a DAG the nodes cannot be trivially grouped into layers -> Neurons in an FFN cannot trivially be grouped into layers. Here, I would say that the way FFNs are constructed is very specific - i.e. one where this mapping from neurons to layers is trivial. That's because an FFN is not a general DAG but a very regular kind of DAG, right? If not what am I missing? $\endgroup$
    – Chillston
    Feb 10 at 20:39
  • 1
    $\begingroup$ You are indeed in the right path. The way you describe it, is exactly what mean. I will consider to complete the question with your remarks. $\endgroup$
    – neoglez
    Feb 12 at 11:16
  • $\begingroup$ One thing that evades me a bit still the underlying reason for the question. Lets say you have a case of an ANN where a mapping from individual neurons to layers is not trivial. (When you think about it, maybe even the transformer is such a case because of the interactions of encoder and decoder and residual connections?). I cannot yet see why thinking about this neuron-to-layer mapping is useful. What would such a mapping be useful for, and, what conclusions could you draw if this mapping is non-trivial? I hope its clear what I mean. $\endgroup$
    – Chillston
    Feb 15 at 18:26
  • $\begingroup$ Well, regarding Q. No. 1, one could ask the same for every aspect of research and science, especially in graph theory and topology. The 2. Q. is a little more perturbing because it uses the argument of utility (Utilitarianism?) to establish the importance of the matter. Philosophical discussions apart, my motivation in asking this question is rather banal. I am writing my dissertation and I realized that I was unable to connect the arguments fluently. The details would need several pages, so I will stop here. I almost found the answer, but I have to double-check it before I post it. $\endgroup$
    – neoglez
    Feb 16 at 17:21
  • $\begingroup$ Sure, the main reason why I was asking that is that a questions sounds a bit confusing if such details aren't clarified (i.e. that this is an explorative question with yet unknown utility). I wish you success with the diss and have my fingers crossed for a positive outcome of this investigation. Also I am actually curious about your argument and in case you find some interesting properties regarding the layering question, maybe you can drop a link or summary ;) $\endgroup$
    – Chillston
    Feb 16 at 18:52

1 Answer 1

Reset to default
1
$\begingroup$

In a practical point of view, a "layer" in a neural network is a step in computation which only depends on its preceding layers' outputs. Note that layer $n$ might take inputs from any previous layers, not only from $n-1$. Those are called skip connections. But since you are an expert, I assume you know all this already.

Usually NNs are defined as a stack of layers, which define a DAG. But given a DAG, there are several valid layer-by-layer interpretations of its structure. But in general, I don't think it is important to know whether a given node is from layer $i$ or $j$. May I ask, what is the motivation behind this question?

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks for the answer. As you stated, I am aware of all the facts you mentioned. $\endgroup$
    – neoglez
    Feb 10 at 17:30
  • 2
    $\begingroup$ @neoglez Ok, but then it's hard to believe that you're an expert and you are still confused about the concept. If this answers your question, you should accept it. If it doesn't then you should ask for clarifications. What's still confusing? $\endgroup$
    – nbro
    Feb 11 at 10:29
  • $\begingroup$ @nbro Please note, that the discussion ist not about whether I am expert or not. If you cerefully read the information you linked to, you can see that an answer can be accepted, when the answer satisfies the question, which ist unfortunately not the case. $\endgroup$
    – neoglez
    Feb 12 at 11:22
  • 1
    $\begingroup$ @nbro well, we can be experts in any field - relatively speaking. But usually an expert don't call himself one, in an believe-it-or-not tone. Beside, we should be careful when spelling the word "cerefully". $\endgroup$
    – lpounng
    Feb 13 at 9:41

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .