1
$\begingroup$

I haven't been able to find a good discussion specifically comparing the two (only one describing a classification and regression problem). I am training a classifier to learn both age and gender based on genomic data. Every sample has a known age and known gender (20 classes in total).

Currently, I am using a single neural network with a sigmoid activation in the last layer with a binary_crossentropy loss. This works fine. However, I also see people using multi-head neural networks where, for example, a set of shared layers would split in to two either additional dense layers or in to two final layers for classification – each with an independent loss (in my case likely a categorical_ce).

What I am unsure of, though, are the advantages and disadvantages between the two (maybe advantages and disadvantages are not the right words to use – actual differences between the two might be more appropriate and when one might use one of those over the other)?

I want to be able to calculate the usual metrics – TP, FP, etc. after training – presumably it would be easier with two heads at the end of the network, as you can work with two independent sets of predictions to calculate these?

$\endgroup$
2
+50
$\begingroup$

If I understood things correctly: You have a task which you need to estimate two values, gender and age. Your question revolves about the difference between networks which share layers for both inputs, whether the shared layers should be followed independent linear layers.

Firstly, using shared layers in the networks of two related tasks may be useful to create more general latent representations in the network hidden layers. It also can speed up training, the shared layer will learn useful features more quickly than if there were two separate networks for each task. Some examples which demonstrate the potential benefit of shared networks can be found in the papers for two RL algorithms, A3C and PPG (PPG adds some extra tricks to the shared layers it):
http://proceedings.mlr.press/v48/mniha16.html
https://arxiv.org/abs/2009.04416v1

Whether the shared layers should be followed by many separate linear layers or a single one, at least for me, isn't something easy to deduce. Intuitively, having a single linear mapping after the shared layers will help prevent over-fitting because the shared layers induce more general features in those layers. While having many separate layers may be useful if there is some complex non linear mapping between the final output and the latent features from the shared layers.

I think the best way to find out is just to experiment with it and see which gives the best results.

A little bit anecdotal, but, an example from my experience:
Shared layers are commonly used in RL for actor critic algorithms. A network takes an image as input and outputs an action and a value (the output for the actor and the critic, receptively). Generally a single linear mapping from the shared layers works just fine, even better than more complex networks.

=========== Edit
In pseudo-code, this is the networks that came to my mind:

# Network 1  
 shared_layer = Linear(input_dim, latent_dim)  
 output_layer = Linear(latent_dim, m + n)  

# Network 2  
 shared_layer = Linear(input_dim, latent_dim)  
 output1_layer = Linear(latent_dim, m)  
 output2_layer = Linear(latent_dim, n)  

And, in the question you mention about using BCE with Network 1 and changing to CE for one of the outputs of the Network 2.
The networks themselves are equal to one another. One implementation might be more practical than the other, but they are the same. Depending on the framework you use, you can use either BCE or CE loss in Network 1 and 2. In Network 1 this would mean taking the output of the last layer and slicing the outputs for age and gender into two variables and applying a loss function in each of them.

That said, I would expect to see a difference between using CE or BCE for age classification. When training with BCE it's possible that one estimator turn to be more 'optimist' or 'pessimist' and gives overall high/low probabilities (this will depend on factors such as whether there is class unbalance, if those are taken into account in the training procedure, etc...). And, that will mean that when you take the maximum probability of the age outputs there will be some bias.
CE seems to me to be a more appropriate choice for age classification. Using CE will not prevent bias if there is class unbalance in your data, but with it is more straightforward to handle these issues.

$\endgroup$
5
  • $\begingroup$ Thank you, my question more more about the difference between shared layers and a 2 separate (as in the case with A2/a3c and one shared layers with a final sense layer of number of outputs (not multi head) and each label being treated as binary, so two neurons would fire in the last layer, one for organ and one for gender. Intuitively I am struggling to see the true difference between the two.. especially if layers are shared upstream which is definitely the case for me. I do have an architecture of shared layers for the reasons you mentioned. $\endgroup$ Jun 6 at 7:19
  • 1
    $\begingroup$ @user9317212 Made an edit to the answer! $\endgroup$ Jun 6 at 13:12
  • $\begingroup$ thanks! for network1 why would you need to apply two losses when the bce loss handles each class as a binary? and therefore can assign to each sample both an organ and an age classification? $\endgroup$ Jun 6 at 15:07
  • 1
    $\begingroup$ You can apply BCE loss for the task. If you have the exact age of your samples you may also treat the problem as a regression problem, and postprocess the estimated value if needed. All these alternatives should work. Which option is better? BCE loss may be a worse alternative to CE for the reason given in the answer (also, a regression loss can be an even better alternative because it provided a richer feedback). Practically speaking, it is difficult to say whether there will be a significant difference in performance. You may test it to find out! $\endgroup$ Jun 6 at 15:58
  • $\begingroup$ Thank you! Will try all methods (it is strictly a classification though and I don't need to do a regression for the purposes of this) in also using interpretation tools to find out why the classification decisions are as they are. $\endgroup$ Jun 6 at 16:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.