How do I assign a weight to an additional loss?

Question

I am trying to do multi-spectral image fusion. I am using the following paper as a reference.

https://arxiv.org/pdf/1804.08361.pdf

The code available on GitHub works well. But, I am trying to add some of my own code. I am trying to add an additional loss to the total loss calculation. The current loss function is sum of two losses, I am trying to introduce a third loss.

I am not sure how do I assign a weight to this additional loss? Because I have read many papers like this one and no one has really gone in depth of how did they came up with the weights assigns to individual losses in case of multi-loss function.

Is there any logical way to solve this problem? I would rather not do trial and error.

Answer 1

Well, it depends on the "meaning" (or expected impact) that the additional loss term has (should have).

For example, there are cases in which minimizing all the losses is detrimental for the network: in VAEs fully minimizing the KL may prevent the decoder to reconstruct the inputs at all. And in some other in which the additional loss term is either a regularizer or a property.

To summarize:

• If all the terms have the same importance (and so they should be all equally minimized), you should weight them such that to have the same scale. Intuitively you want to gradient of each loss term to have almost the same magnitude.
• Otherwise, if the other term is either a regularizer or something that have to ensure a property you need to search for the $\lambda$ that best weights the extra term. This is because usually such term should have a limited contribution on the overall loss. One practical way to do so is to apply grid search first to find the magnitude of $\lambda$, e.g. considering a scale within $[10^{-2}, 10^{-1}, 1, 10, 10^2]$. Then use either a random search or some more sophisticated hyperparameter tuning algo (like bayesian optimization) within the interval that grid search found, e.g. between $[10^{-1}, 1]$.

Answer 2

Sure, you can definitely add your own loss term to the originally computed loss function. I checked the repo of your paper, they have used tensorflow. I am giving you a minimum reproducible example of how can you make your own custom loss function and train your model with the computational graph of tf.

For more details you can check this link to other question on stack. link

import tensorflow as tf

# Assuming you have some model defined and the inputs and targets are placeholders or tensors
inputs = ...
targets = ...

# Define your loss function
def my_loss_function(predictions, targets):
    # Calculate the additional loss term
    additional_loss = ...

    # Calculate the main loss (e.g., mean squared error)
    main_loss = tf.reduce_mean(tf.square(predictions - targets))

    # Combine the additional loss term with the main loss
    total_loss = main_loss + additional_loss

    return total_loss

# Calculate the predictions of your model
predictions = model(inputs)

# Calculate the total loss by calling your custom loss function
loss = my_loss_function(predictions, targets)

# Perform backpropagation and update the model parameters
optimizer = tf.train.AdamOptimizer()
train_op = optimizer.minimize(loss)