How to build a DQN agent which can be trained through interactive learning?

Question

I am trying to create a chatbot whose dialogue policy model will be trained through reinforcement learning. Dialogue Policy is responsible for selecting the action to take based on the given state of the conversation.

All implementations I see for RL are trained from an environment taken from Gym or created manually. These environments provide the next state, rewards etc to the model based on which it is trained.

Since I am creating a dialogue policy model which will be trained through real user conversations, I cannot provide a "pre-defined" environment which can provide the states and rewards. I am planning to train it myself by talking to it and providing rewards and next state (which I think is called interactive learning).

But I was not able to find any implementations, tutorials or articles on interactive learning. I am not able to figure out how to create such a model, how to take care of the episodes, sessions etc. This will be a continuous learning that will go on for months maybe. I have to save the model each day and continue training the next day by loading the model from that same state.

Can anyone guide me in the correct direction on how to approach this? Any githubs links, articles, tutorials of such implementations will be highly appreciated. I am aware this question seems too broad, but some hints will be very helpful for a newbie like me.

Answer 1

I suggest you to start reading this one on BERT and this one on GPT-2.

I am aware this question seems too broad, but some hints will be very helpful for a newbie like me.

I'm not sure you want to create your chatbot using RL architecture at all. But if you want to implement such an idea the right way to go with that is by using iterative approach and starting from a really simple base:

1. Define a list of actions for your agent. Might be 10-words vocabulary with simple one-word answers.
2. Define you simple environment. It might have 20 different states (e.g. greetings/conversation starters hey, hello)
3. Map good responses (hello -> hi) to positive rewards and -1 for the rest of them.
4. If you want to move forward with DQN try a simple Q-learning algorithm on your simple environment and train agent to properly response to those.
5. Once you have done the above you can make your vocabulary bigger
6. Now replace your table of input words which gives you a state for your network with NN/word2vec/any other NLP approach which will convert your input sentence to a state.
7. Manually provide a reward as a response for what you agent did according to learned policy.
8. Do that infinite amount of time and your agent probably will figure out how to respond properly.

NOTE: DQN uses discrete action space. So in such a case you will have only limited or REALLY HUGE actions space. It's all possible combinations between all possible words in you vocabulary (assuming action maps to some sentence)