4
$\begingroup$

I find the logistic map absolutely fascinating. Both in itself (because I love fractal) and because it is observed in nature (see: https://www.youtube.com/watch?v=ovJcsL7vyrk).

I'm wondering if anyone tried it as an activation function in some way or another with any kind of success.

I like it because it has some kind of "I'm not sure what to do" above ~3.0 and the less confidence the more chaotic the response is. It gives the possibility to explore some other solution to escape a local optimum (not sure I use this word correctly). And below 3 it's still a nice and smooth activation function like, eg, a tanh.

Eg : the reward i got isn't the reward i expect, and the higher the difference the more i'll explore other solution. But it's still gradual, from 1 choice, to 2 choice, 4, 8, 16, ... until it become chaotic. (giving the possibility to experiment some pseudo-random choice). And below this threshold it still act as a usable "good old" activation function.

Another good side is that it's gpu-friendly and don't need many iteration for this application since a little bit of uncertainty (even below the threshold) isn't undesirable. see : https://upload.wikimedia.org/wikipedia/commons/6/63/Logistic_Map_Animation.gif

Edit : so, ok, i tested it on my extremely naive racetrack. (feedforward, no feedback, no error, no fitness, only genetic selection for the car that didn't crash). It does work, for sure. I don't see any advantage in practive but with such a naive NN, there isn't much i can tell.

My implementation :

def logi(r):
    x = .6  # the initial population doesn't matter so i took .6
    for _ in range(random.randrange(10,40)):
        x = r * x * (1 - x)
    return x

The activation take 8% of my laptop cpu (while is was invisible on my radar with leaky leru)

Logistic map bifurcation diagram

$\endgroup$
19
  • 1
    $\begingroup$ Can you explain more in detail how you would use the "logistic map" as an activation function? Maybe it would be a good idea if you briefly describe what the "logistic map" is, how it is defined, and why you think it could be used as an activation function. $\endgroup$ – nbro Nov 23 '20 at 17:25
  • $\begingroup$ i'm not quite sure, i'm still struggling with my code and don't have the leisure to experiment like this, yet. But the general idea is to allow the NN to have some level of chaotic behavior. Eg : the reward i got isn't the reward i expect, and the higher the difference the more i'll explore other solution. But it's still gradual, from 1 choice, to 2 choice, 4, 8, 16, ... until it become chaotic. (giving the possibility to experiment some pseudo-random choice). And below this threshold it still act as a usable "good old" activation function. $\endgroup$ – ker2x Nov 23 '20 at 23:22
  • $\begingroup$ The formula is still simple ipython-books.github.io/… but a bit slow since it is an iterated function. $\endgroup$ – ker2x Nov 23 '20 at 23:24
  • $\begingroup$ Another good side is that it's gpu-friendly and don't need many iteration for this application since a little bit of uncertainty (even below the threshold) isn't undesirable. see : en.wikipedia.org/wiki/Logistic_map#/media/… $\endgroup$ – ker2x Nov 23 '20 at 23:30
  • $\begingroup$ I suggest that you edit your post to include more details that explain your idea. $\endgroup$ – nbro Nov 23 '20 at 23:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.