Are there (complex) tabular datasets where deep neural networks (e.g. more than 3 layers) outperform traditional methods such as XGBoost by a large margin?

I'd prefer tabular datasets rather than image datasets, since most image dataset are either too simple that even XGBoost can perform well (e.g. MNIST), or too difficult for XGBoost that its performance is too low (e.g. almost any dataset that is more complex than CIFAR10; please correct me if I'm wrong).

Thank you!

  • $\begingroup$ I just remembered a good example... blog.kaggle.com/2012/11/01/… but that was in 2012 and didn't outperform "by a large margin" and was before the xgboost mania. $\endgroup$ – user1269942 Dec 12 '19 at 20:27
  • $\begingroup$ I'll try out the dataset used in this post: it's great that the columns are uncleaned, since I'd expect the advantage of DNN to be better pronounced when the features are less structured. Thank you @user1269942! $\endgroup$ – Clara Dec 14 '19 at 20:54

In my opinion, no. Also images could be interpreted as tabular dataset as well, where certain columns represent different rgb codes of pixels. If you seek to use neural nets opt for image datasets, with large sample size. Neural networks generally require large sample sizes to perform, and huge dimension inputs to not be outperformed by boosting.

  • $\begingroup$ That makes a lot of sense, thank you Matthew! May I ask if you know any complex / large tabular dataset (e.g. perhaps with low-level features as columns) that could be difficult for boosting? Thank you! $\endgroup$ – Clara Dec 11 '19 at 15:59
  • $\begingroup$ Unfortunately, I don't. But you can always make one for yourself, out of image data sets, $\endgroup$ – MatthewTuby Dec 12 '19 at 7:25
  • $\begingroup$ I can confirm this anecdotally. I have tried many times to apply NNs to tabular and/or non-complex data and I cannot recall a NN ever doing better than more "traditional" models. Ha ha, of course, we all know that the plural of anecdote is not data! So exceptions may exist. I do, however, disagree with trying to represent an image as column data. $\endgroup$ – user1269942 Dec 12 '19 at 20:18
  • $\begingroup$ Disagree on what ground? Possibility or usefulness? $\endgroup$ – MatthewTuby Dec 13 '19 at 7:38
  • $\begingroup$ @MatthewTuby on the grounds that it will likely make the typical convolution operators ineffective as "columnified" image data will likely lose the benefit that pixels near-by in the actual image are near-by in the new representation(column data) like they are in the standard RGB layered matrix representation. A pedant could assert that the normal matrix representation is constructed from columns...but that is not what is being implied here. One example of image data in column format, used in GIS, is the .csv format with encodes a single channel image into "x,y,value" rows. $\endgroup$ – user1269942 Dec 15 '19 at 21:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.