2
$\begingroup$

The Kohonen network is one fully connected layer, which clusters the input into classes by a given metric. However, the one layer does not allow to operate with complex relations, that's why deep learning is usually used.

Is it possible then to make multi-layered Kohonen network?

AFAIK, the output of the first layer is already cluster flags, so the activation function on the non-last layers must be different from the original Kohonnen definition?

|improve this question
$\endgroup$
1
$\begingroup$

Kohonen networks by definition are single layer FCNN's, but what differentiates them from others is their unsupervised training procedure.

This procedure is a function of the input, the weights and some hyperparameters. This means if you have a multilayer network you could train only the final layers weights using this procedure. Think of it this way, let $f$ be the final layer, and $g$ be the composition of all other layers, such that the whole network $N(x) = f \circ g(x)$. Using kohonens training procedure, you could learn $f$'s weights where the input is $g(x)$ rather than x. But then how would you learn $g$'s weights?

So you could do this by iterative learning. let $N$ be a 2 layer network: $N = f_2 \circ f_1$. first learn $f_1$ using the single layer procedure, and now you could learn $f_2$ in the same manner, except the input features its trying to cluster would be $f_1(x)$ rather than $x$.

There does exist cons to this procedure though: Using a kohonen network at each layer will try to cluster it to the best of its abilities, not so that its final composition will be best (a common problem in many optimization procedures that arent end-to-end), but that its current representation is. This may lead to non-optimal results, not achieving that deep representation that you're looking for that you could achieve in autoencoders or other deeper unsupervised models.

|improve this answer
$\endgroup$
  • $\begingroup$ What did you mean by learn $f_1$. Afaik, in ideal cases kohonen is supposed to output a spike in a single output node, so of we learn this and put another layer on top, I don't see any kind of advantage, unless we do incomplete learning where the spike is smooth and more of a hill, but then again I am unsure how that'll provide any extra help. $\endgroup$ – DuttaA Jul 17 at 19:08
  • $\begingroup$ I also guessed you can not use kohonnen with the normal layers, because the mertic of those output is not like the from original data, means it is not differomorf. I v made small test and watched that for example in autoencoders middle layer. I v asked, is it generally known multiple knns -> means not. $\endgroup$ – user8426627 Jul 17 at 19:22
  • 1
    $\begingroup$ @DuttaA thats kinda my point: since you can learn f1 then f2, f2 will add almost no benefit because its optimized for the singular goal at each layer rather than for a deeper end-to-end representation you get in autoencoder type models $\endgroup$ – mshlis Jul 17 at 22:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.