0
$\begingroup$

I fed the same set of 1.4 million data to two different models:

  1. MLP
  2. CNN model

In both cases, I used the same parameters and hyperparameters.

The CNN is showing comparatively lower accuracy (80%) than that of the MLP (82%).

Why?

And, also, what does this experiment tell us?

Edit:

Is the data images, video or audio, or other grid-based signals (same signal repeated at multiple locations with a meaningful distance metric between them) in your case?

It is protein C-alpha distances data that has 3 classes (helix, strand, coil) and n-number of features, where n is an even number.

In fact, what is it, what was your expectation of the relative performance of the two models, and why?

I thought CNN would be more efficient and thereby would demonstrate better test/validation accuracy.

$\endgroup$
6
  • $\begingroup$ Is the data images, video or audio, or other grid-based signal (same signal repeated at multiple locations with a meaningful distance metric between them) in your case? In fact what is it, what was your expectation of relative performance of the two models, and why? $\endgroup$ Sep 21, 2021 at 6:20
  • $\begingroup$ @NeilSlater, see the edit. $\endgroup$
    – user366312
    Sep 21, 2021 at 6:50
  • $\begingroup$ Thanks for the edit. Sorry, I still don't understand your description of the data. A protein could be represented by a 1d signal of identical features (e.g. an array of amino acids), but could be anything else. I need more. E.g. If I said my data was "car data" you would have no clue whether it was pictures of cars, measurements of dimensions, engine stats, accident stats etc. Please give something more useful to work with. Also, while I am asking, how have you "balanced" between the models in terms of number of parameters (perhaps give the basic number of neurons etc). $\endgroup$ Sep 21, 2021 at 7:03
  • $\begingroup$ @NeilSlater, C-alpha distances. $\endgroup$
    – user366312
    Sep 21, 2021 at 7:23
  • $\begingroup$ So you are doing something like this: researchgate.net/publication/… ? $\endgroup$ Sep 21, 2021 at 9:02

1 Answer 1

2
+50
$\begingroup$

To get a full understanding of your problem, one would like to know what approximately the $n$-features are.

Whether, it is about the geometrical structure, protein is described by a graph, where vertices correspond to atoms and edges to bonds within them - I would consider use of GraphNN, there is some research, that has demonstrated the success of GraphNN for protein prediction:

In case, your data is some general form tabular data with features of different nature and form - like some continuous features, binary features, categorical features, whatever, there is no notion of proximity between different features.

The efficiency of CNN is based a lot on the ability of them to have a notion of locality, aggregate the information from the neighborhood, and construct the hierarchy of low-level (for several neighboring pixels) features and global (that understand the image or a signal as a whole) features.

Two neighboring pixes from the cat's ear have a notion of proximity and relatedness, whereas the red color and square shape do not have.

For unstructured tabular data I would recommend to start from some tree ensembling approach:

  • Random forest
  • Gradient boosting

EDIT

Seems like what you receive as input is a matrix of pairwise distances $\rho(i, j)$ between all possible amino residues - Protein contact map.

I would think about it as a weighted adjacency matrix. However, there is a rather special structure, enter image description here

Therefore, there is no need for generic GNN, most likely.

In the literature, the recent research solves this problem with the help of CNN:

Probably, you should tune some hyperparameters, or introduce residual connections, if there are no such at the moment.

$\endgroup$
2
  • $\begingroup$ What do you mean by residual connection? $\endgroup$
    – user366312
    Sep 27, 2021 at 23:34
  • $\begingroup$ @user366312 for deep neural networks - with the number of layers more then 10, say, networks become very hard to optimize and accuracy saturates at rather high loss (train and test). I was a breakthrough, when Resnets were introduced, where after each layer, without downsampling or change of number of channels, one adds the input of the stack of layers to the output. Concatenation is possible as well. This combats the problems with optimization. $\endgroup$ Sep 28, 2021 at 4:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .