2
$\begingroup$

I'm writing a DQN agent for the Wumpus game.

Is the reward function to train the Q-networks (target network and policy) the same as the score of the game, i.e. +1000 for picking up gold, -1000 for falling in pits and dying from the wumpus, -1 each move?

This is naturally cumulative, in that the score changes after each action taken by the agent. Alternatively, is it just a +1 for win, -1 for a loss and 0 in all other situations?

$\endgroup$
1
$\begingroup$

The reward function is up to you when you set the goals for the agent.

  • If the goal is to score as highly as possible, before ending the game, then use the score. You may want to scale the score down if you are using neural networks, to prevent needing to handle very large error values in early phases of learning.

  • If the goal is to win the game, and you do not care about the score, then use the win/loss end result. I am not familiar with the game, but if it is possible to win the game - e.g. reach an exit - whilst not collecting all the gold, then the agent may choose to do that if it reduces the chance of losing.

The second option is harder for the agent to assess. You may want the current score to be one of the state variables, as the score is likely to be correlated with win/loss.

Most computer games are designed around giving a numerical score as feedback for human play, with high score tables, players considered "better" if they get more points etc. If you want your agent to compete in the same way, then using the score directly will help achieve that goal.

| improve this answer | |
$\endgroup$
1
$\begingroup$

The reward function belongs the the environment and it is the only way the agent can explore the world given a state.

If we want agent to do something specific, we must provide rewards to it in such a way that it will achieve our goals. It is thus very important that the reward function accurately indicates the exact behaviour.

Depending on your goal you can construct the function such that the agent will try to finish the game as fast as possible, or collect the maximum score.

For example, certain reward functions can cause an agent to commit suicide in order to avoid more severe punishment in form of negative reward in the future (e.g. if the step reward very small). Or it will go the safest way without collecting gold, if falling in pits punishment is very big. In other words, you should experiment with your reward function to find a tradeoff.

Check out this video for more intuition behind it.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.