Timeline for Initialising DQN with weights from imitation learning rather than policy gradient network
Current License: CC BY-SA 4.0
14 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Nov 15, 2020 at 5:58 | vote | accept | calveeen | ||
| Nov 14, 2020 at 17:53 | answer | added | Neil Slater | timeline score: 1 | |
| Nov 14, 2020 at 16:02 | comment | added | nbro | @NeilSlater Hm, well, it's true that if you have only a dataset of state-action pairs (i.e. without rewards and transitions), you cannot learn a value function (which is what I think you mean, though, in my mind, I hadn't restricted IL to those datasets, but also to any supervised way of learning a policy or value network with a dataset, which can be composed of just state-action pairs, or maybe it's a dataset like an experience replay, i.e. with transitions $(s, a, r, s')$). Now, the question is then: how can you initialize the value network with the policy network's weights? | |
| Nov 14, 2020 at 15:56 | comment | added | Neil Slater | @nbro: It is not possible to learn a value network via imitation learning, at least not by strict definitions. The data could be used in off-policy reinforcement learning - my deleted answer attempts to address what could go wrong there. However, I think I have now clarified that the OP is not doing that. Instead they have trained a policy network through imitation learning, and want to take advantage of that learning to set up a value-based method | |
| Nov 14, 2020 at 15:31 | comment | added | nbro | To me, it's not clear yet what you obtain from the imitation learning phase, when you say "I would like to use weights learnt from the imitation network". In other words, is "imitation network" a value network (and not a policy) that you learn by imitation learning, right? Of course, this should be the case, but I just wanted you to clarify this. | |
| Nov 14, 2020 at 15:19 | comment | added | Neil Slater | Could you clarify what the value range can be in your problem, and whether the value-predicting network uses any non-linearity in the output layer? Typically a DQN uses linear output layer - is that the same in your case? | |
| Nov 14, 2020 at 14:50 | comment | added | calveeen | sorry i have edited to make the question a little more clearer | |
| Nov 14, 2020 at 14:46 | history | edited | calveeen | CC BY-SA 4.0 |
added 88 characters in body
|
| Nov 14, 2020 at 14:18 | comment | added | nbro | I suppose a value network, otherwise, not sure how you would transfer the policy's weights to a value network. Moreover, it may also be a good idea to explain more in detail what you mean by "imitation learning", i.e. which imitation learning technique are you using. It's also not clear what reward shaping has to do with your question. | |
| Nov 14, 2020 at 14:12 | comment | added | nbro | @calveeen Please, clarify what your main specific question is. Right now, that's not clear. Do not ask "Might anyone have an opinion on this?", i.e. do not ask for opinions, but ask a question that can be answered objectively. Is your question: "Can we initialize the weights of a value network (or DQN) with the weights of another value network trained with imitation learning? Why doesn't this seem to work for me?" It's not clear, from your description, what exactly are you training by imitation learning (i.e. supervised learning). Is it a value network or a policy? | |
| Nov 14, 2020 at 14:11 | history | edited | nbro | CC BY-SA 4.0 |
added 7 characters in body; edited tags
|
| Nov 14, 2020 at 12:54 | comment | added | calveeen | Hi Neil, yes i am training an imitation network using supervised learning first and then transferring the weights over to a value network. (The authors in alphaGo used a policy network instead). I am contemplating whether the transfer of weights would work for a value network rather than for a policy gradient network. The rewards in the problem uses sparse rewards which is observed only after long time steps, which is why I am using imitation learning to learn a good initial policy first. | |
| Nov 14, 2020 at 12:38 | comment | added | Neil Slater | Could you clarify the training and transfer process? Are you training a policy network using imitation learning then copying the weights over to a value-predicting network? If you are doing so, could you clarify what the value range can be in your problem, and whether the value-predicting network uses any non-linearity in the output layer (the policy network presumaby uses softmax)? | |
| Nov 14, 2020 at 10:52 | history | asked | calveeen | CC BY-SA 4.0 |