3
$\begingroup$

Interested to know if there was any use or interest in activation functions with more than one output value to the next column instead of single firing.

I'm interested to know if this would have any use or would be almost identical to a single value firing.

enter image description here

|improve this question
$\endgroup$
  • 1
    $\begingroup$ How exactly can it have multiple firings? Although it might be useful depending on how you are implementing it (I have never heard of multiple firings) but the result can still be achieved by single firing by just increasing the number of hidden nodes $\endgroup$ – DuttaA Mar 13 '18 at 16:26
  • 1
    $\begingroup$ How would the new multiple values be decided? Could you give any example worked in more detail? For instance, a single value output is usually decided as e.g. tanh( sum(input*weight) + bias ). Are you intending to have multiple different functions, some additional multipliers - if so are they flexible, and how etc? $\endgroup$ – Neil Slater Mar 13 '18 at 19:20
  • $\begingroup$ My thinking is in a similar way the network might be initialised with random weights to the incoming nodes, the outgoing firings may start with random weights which also get adjusted over the learning process $\endgroup$ – benbyford Mar 15 '18 at 13:48
2
$\begingroup$

I think this is an interesting approach, but it's hard to tell if it can make better or faster learning networks.

If I understand correctly, this could be looked at as using a function returning a tuple as activation function. The number of elements in the tuple must be equal to the number of neurons in the next layer.

For example you could use an activation function like f(x) = (ReLU(x), sigmoid(x)) if the next layer contains two neurons.

I think the biggest problem is to determine why one value should be passed to exactly THAT neuron and another value to another neuron - why not opposite? This must either be determined randomly or by doing some advanced analysis in beforehand.

Anyway, I think it's an interesting idea which should be tested out, but it's hard to say whether it will give better or worse results before trying it out.

|improve this answer
$\endgroup$
  • 1
    $\begingroup$ Thanks @mr-eivind. not sure if it would be good or bad either, but just thought it was something worth thinking about $\endgroup$ – benbyford Aug 17 '18 at 14:31
1
$\begingroup$

It's not really clear what you are asking here.

If you mean that a neuron in one layer can have connections to many neurons in the next layer, then this is already the standard topology. In fact, it's the only one that's interesting.

If you mean that a neuron in one layer can be "on" (firing) for some neurons in the next layer, but "off" (not firing) for other neurons in the next layer, then this is also already a standard feature: setting the weights between the layers appropriately (i.e. to 0) will cause a given neuron to appear off. Setting the weights to other values will determine the degree of signal passed to each neuron in the next layer.

So I think the answer to your question is that it would be identical to a "single" firing.

|improve this answer
$\endgroup$
  • $\begingroup$ Maybe I miss communicated or missunderstand. I was wondering if each neuro passes on a single firing or value to each of the next layer or whether it could send a specific firing to each neuro it was linked to $\endgroup$ – benbyford Jul 24 '18 at 9:46
  • $\begingroup$ As explained in the answer, the weights between a neuron and each neuron in the next layer are allowed to be different, and scaling those weights allows arbitrary signals to be sent. $\endgroup$ – John Doucette Jul 24 '18 at 12:36
  • $\begingroup$ @benbyford That's precisely what takes place in normal models. $\endgroup$ – Daniel Aug 15 '18 at 22:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.