Maxout networks were a simple yet brilliant idea of Goodfellow et al. from 2013 to max feature maps to get a universal approximator of convex activations. The design was tailored for use in conjunction with dropout (then recently introduced) and resulted of course in state-of-the-art results on benchmarks like CIFAR-10 and SVHN.

Five years later, dropout is definitely still in the game, but what about maxout? The paper is still widely cited in recent papers according to Google Scholar, but it seems barely any are actually using the technique.

So is maxout a thing of the past, and if so, why — what made it a top performer in 2013 but not in 2018?

  • 2
    $\begingroup$ Then research it further, and you may be the next Goodfellow. $\endgroup$ – Douglas Daseeco Jul 11 '18 at 3:15

Basically, if you read the full paper (especially, the abstract and the section 7), you find that the main accomplishment remains a marginal contribution on top of dropout.

If you see the empirical results on Table 5 (of the page 5) of the maxout's original paper, you find that the misclassification rate is only very, very slightly lower than that of dropouts. (2.47 % instead of 2.78%)

That could explain the relatively lower interest in the work.

  • $\begingroup$ Thanks for your answer. Cutting the error from 2.78% down to 2.47% is actually a 11% decrease in the error rate. Not a game changer but still interesting and definitely not tiny. $\endgroup$ – user209974 Mar 18 '19 at 10:47
  • $\begingroup$ Fair point. If that is of interest to your specific application, it certainly is worth investigating. $\endgroup$ – Amrinder Arora Mar 18 '19 at 15:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.