2
$\begingroup$

I have a combined network consisting of two parts: one is for images and the other is for numerical data. Each sample is matched with a numerical case by an ID. For this combined network, a lr of 0.01 was found to be best working via hyperparameter tunings

However, when I trained them as a separate task (binary classification), a lr of 0.001 for images and 0.01 for numerical data were best. As for an AUC metric, the combined network (0.818) is performing on average of image and numerical networks (0.799 and 0.821, respectively). Here I thought maybe the combined network's lr is too high for image part and should apply lower lr for that part. But, I don't know if it is possible.

If anyone has any idea of what is what, let me know

Improve this question
$\endgroup$
2
  • $\begingroup$ Is this a programming issue/question or a conceptual/theoretical one? You tagged your question with PyTorch, so maybe you're looking for a solution in PyTorch and not really for a conceptual answer. If yes, then this question is off-topic here, and should be asked on Stack Overflow, because it would be a programming issue, which is off-topic here. Please, let me know, because this post was voted to be closed as off-topic, probably because of that. $\endgroup$
    – nbro
    Jan 14, 2021 at 16:43
  • $\begingroup$ it is a conceptual one, I tagged pytorch just in case someone has done something similar in pytorch $\endgroup$
    – bit_scientist
    Jan 14, 2021 at 23:47

1 Answer 1

Reset to default
1
$\begingroup$

Create two different optimizers and split the subnets' parameters into either with different lrs. You will have to call optimizer1.step(), optimizer2.step() with a single backward() call

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .