6
$\begingroup$

As I know, the current state of the art methods for training deep learning networks are variants of gradient descent / stochastic gradient descent.
What are the best known gradient-free training methods (mostly in visual tasks context)?

$\endgroup$
  • $\begingroup$ Please comment to explain downvotes. $\endgroup$ – rursw1 Aug 24 '17 at 12:44
  • 2
    $\begingroup$ As a newbie here, I experienced similar thing. As I see it, down-votes are a bug like a feature in the democratic process. They may have good reasoning or no reasoning about it. It'd be useful if the down-votes reasoning are also audited in similar manner edits are audited. There should be some logic behind why such a downvote was casted. This will help newbies engage in this forum in smarter manner. Also, It'll be helpful if new members know where to find common answers. $\endgroup$ – Rajib Bahar Aug 24 '17 at 13:02
  • 1
    $\begingroup$ (It's possible the downvoters consider "best known" too subjective.) Don't be discouraged though. Initial involvement with most stacks can be rocky, but I think this is a useful question. Welcome to AI! $\endgroup$ – DukeZhou Aug 24 '17 at 18:25
  • $\begingroup$ Several algorithms are mentioned in this slide of class of M.Kochenderfer adl.stanford.edu/aa222/Lecture_Notes_files/… (p.s. The problem is more general then Deep Learning) $\endgroup$ – bruziuz Jan 30 '18 at 13:53
  • $\begingroup$ Generally this days ML community use gradient based methods. Even if they don't fully understand "why they do it" but it's because methods are robust and fast as proved by math scientists. p.s. And non-convex optimization is hard and non well solved. There are a lot of approximate methods how to deal with it - welcome to math optimization. $\endgroup$ – bruziuz Jan 30 '18 at 13:59
3
$\begingroup$

There are several different algorithms that can be used for gradient free neural network training. Some of these algorithms include particle swarm optimization, genetic algorithms, simulated annealing, and several others. Almost any optimization algorithm can be used to train a neural network. Here is an overview of some of the algorithms I listed:

  • Particle Swarm optimization - I would say that this is one of the better optimization algorithms to train neural networks other than back propagation. I am currently using it and have achieved quite good results.
  • Genetic Algorithms - I have tried to use genetic algorithms to train neural networks in the past and I was not able to get it to work. However, I was using deep neural networks with almost a million parameters and the performance was not that good.
  • Simulated annealing - simulated annealing is based off of metals cooling. I have seen simulated annealing work fairly well but maybe not as well as particle swarm optimization.
  • Derivatives of genetic algorithms - derivatives of genetic algorithms such as NEAT have been shown to work pretty well. I have not personally used them extensively but some of the things that people have used them for are pretty cool.
$\endgroup$
  • $\begingroup$ Do you have maybe some SW recommendations? (If this is beyond this question, please let me know) $\endgroup$ – rursw1 Sep 24 '17 at 6:50
  • 1
    $\begingroup$ @rursw1 I am currently experimenting with my own custom GPU accelerated CUDA c/C# PSO implementation, but MATLAB would probably be good for this, and there is a library for NEAT. If you want to use my PSO implementation, just let me know. $\endgroup$ – Aiden Grossman Sep 24 '17 at 18:41
  • $\begingroup$ This is very generous, thank you. I'll try to implement a PSO implementation on my own for now. Thanks! $\endgroup$ – rursw1 Sep 25 '17 at 6:24
  • $\begingroup$ @rursw1 if you need anything else. Just PM me. $\endgroup$ – Aiden Grossman Sep 26 '17 at 0:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.