I have a combined network consisting of two parts: one is for images and the other is for numerical data. Each sample is matched with a numerical case by an ID. For this combined network, a lr of 0.01 was found to be best working via hyperparameter tunings

However, when I trained them as a separate task (binary classification), a lr of 0.001 for images and 0.01 for numerical data were best. As for an AUC metric, the combined network (0.818) is performing on average of image and numerical networks (0.799 and 0.821, respectively). Here I thought maybe the combined network's lr is too high for image part and should apply lower lr for that part. But, I don't know if it is possible.

If anyone has any idea of what is what, let me know

  • $\begingroup$ Is this a programming issue/question or a conceptual/theoretical one? You tagged your question with PyTorch, so maybe you're looking for a solution in PyTorch and not really for a conceptual answer. If yes, then this question is off-topic here, and should be asked on Stack Overflow, because it would be a programming issue, which is off-topic here. Please, let me know, because this post was voted to be closed as off-topic, probably because of that. $\endgroup$ – nbro Jan 14 at 16:43
  • $\begingroup$ it is a conceptual one, I tagged pytorch just in case someone has done something similar in pytorch $\endgroup$ – bit_scientist Jan 14 at 23:47

Create two different optimizers and split the subnets' parameters into either with different lrs. You will have to call optimizer1.step(), optimizer2.step() with a single backward() call


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.