3
$\begingroup$

I'm working with deep learning on some EEG data for classification, and I was wondering if there's any systematic/mathematical way to define the architecture of the networks, in order to compare their performance fairly.

Should the comparison be at the level of neurons (e.g. number of neurons in each layer), or at the level of weights (e.g. number of parameters for training in each type of network), or maybe something else?

One idea that emerged was to construct one layer for the MLP for each corresponding convolutional layer, based on the number of neurons after the pooling and dropout layers.

Any ideas? If there's any relative work or paper regarding this problem I would be very grateful to know.

Thank you for your time

Konstantinos

Improve this question
$\endgroup$

2 Answers 2

Reset to default
0
$\begingroup$

Konstantine, I assume you refer to plain MLP and CNN, without any modifications.

I believe what you ask is how to set both of them up, in order to have the fairest comparison possible.

The way I would do it, is to use their plain implementations but both tuned as much as possible, in every hyperparameter. Both should work as black boxes that accept the same inputs and give the same outputs.

This will give you insight on the true raw performance of both algorithms.

Hope it helps :)

Improve this answer
$\endgroup$
2
  • $\begingroup$ I'm not sure what plain implies. My models are sequential models with two hidden layers (for the convolutional network the first hidden layer is composed by two convolutional layers and a pooling layer). The idea of comparing the best out of the two is appealing. Although i don't have a lot of hyperparameters to play with, so I was thinking more of a size comparisons between the two (size in number of neurons, weights, parameters to learn, etc) $\endgroup$
    – Constantine Pat.
    Jun 14, 2018 at 11:24
  • $\begingroup$ By "plain" I refer to the most "basic" implementation of both algorithms. I don't see direct evaluation from just comparing the e.g. weights of both. This is because the architecture is different so you are comparing "apples and oranges". What would make sense is to compare the overall performance, not the inner characteristics. This means, don't focus on comparing hyperparameters, just make sure you fine-tune both networks. You should compare their results for the same inputs. Results like accuracy and training/inference time. Hope it makes sense. Kalh tyxh :) $\endgroup$
    – pcko1
    Jun 14, 2018 at 11:47
0
$\begingroup$

The best best way to monitor an architectures performance would be comparing the resource utilization, model accuracy, value loss and confusion matrix.
e.g VGG16 consumes less system resources in comparison to inception V3

There is an article that goes in depth.

Improve this answer
$\endgroup$
1
  • $\begingroup$ The article talks more about various metrics of performance, which is not really my concern. The idea of resource utilization is interesting. Any ideas how to compare these in my models? (I use the keras library). Perhaps check the number of parameters to train for each network? (keras outputs the total number of trainable parameters) $\endgroup$
    – Constantine Pat.
    Jun 14, 2018 at 11:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .