Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.
Sign up to join this community
I'm writing a DQN agent for the Wumpus game.
Is the reward function to train the Q-networks (target network and policy) the same as the score of the game, i.e. +1000 for picking up gold, -1000 for falling in pits and dying from the wumpus, -1 each move?
This is naturally cumulative, in that the score changes after each action taken by the agent. Alternatively, is it just a +1 for win, -1 for a loss and 0 in all other situations?
The reward function is up to you when you set the goals for the agent.
If the goal is to score as highly as possible, before ending the game, then use the score. You may want to scale the score down if you are using neural networks, to prevent needing to handle very large error values in early phases of learning.
If the goal is to win the game, and you do not care about the score, then use the win/loss end result. I am not familiar with the game, but if it is possible to win the game - e.g. reach an exit - whilst not collecting all the gold, then the agent may choose to do that if it reduces the chance of losing.
The second option is harder for the agent to assess. You may want the current score to be one of the state variables, as the score is likely to be correlated with win/loss.
Most computer games are designed around giving a numerical score as feedback for human play, with high score tables, players considered "better" if they get more points etc. If you want your agent to compete in the same way, then using the score directly will help achieve that goal.
The reward function belongs to the environment and it is the only way the agent can explore the world given a state.
If we want agents to do something specific, we must provide rewards to them in such a way that they will achieve our goals. It is thus very important that the reward function accurately indicates the exact behavior.
Depending on your goal you can construct the function such that the agent will try to finish the game as fast as possible or collect the maximum score.
For example, certain reward functions can cause an agent to commit suicide in order to avoid more severe punishment in form of negative reward in the future (e.g. if the step reward is very small). Or it will go the safest way without collecting gold if falling in pits punishment is very big. In other words, you should experiment with your reward function to find a tradeoff.
Check out this video for more intuition behind it.
