2
$\begingroup$

I use Q-learning in order to determine the optimal path of an agent. I know in advance that my path is composed of exactly 3 states (so after 3 states I reach a terminal state). I would like to know how to include that in the updating rule of the q-function. What I am doing currently 

for t=1:Nb_Epochs-1       
        if rand(1)<Epsilon
            an action 'a' is chosen  at random             
        else
           [Maxi a]=max(QIter(CurrentState,:,t)); 
        end
        
         NextState=FindNextState(CurrentState,a);
         QIter(CurrentState,a,t+1)=(1-LearningRate)*QIter(CurrentState,a,t)+LearningRate*(Reward(NexState)+Discount*max(QIter(NextState,:,t)));
         CurrentState=NextState;
end

$\endgroup$
5
  • 1
    $\begingroup$ Two questions: When you say "I know in advance that my path is composed of exactly 3 states ", is it the optimal path you are referring to or any path the agent can possibly take? How is your reward function defined? $\endgroup$
    – Jeanba
    Dec 17 '19 at 7:56
  • $\begingroup$ Any path should be composed of only 3 states. My reward function is defined as follows Reward(CurrentState,NextState)=Distance(CurrentState,NextState)*Vector(NextState); where Vector associate each state with a constant reward. $\endgroup$ Dec 17 '19 at 8:04
  • 1
    $\begingroup$ The known length of the path might be used in the update rule some special circumstances, but usually it is not, even if fixed. So, is your question basically the title - i.e. what the update rule should be when the NextState is terminal? Or is your question about special things you could do because you know the trajectory length is always 3? $\endgroup$ Dec 17 '19 at 8:23
  • $\begingroup$ Yes, my question is what is the update rule when my state is terminal ? :) $\endgroup$ Dec 17 '19 at 8:26
  • $\begingroup$ I closed this as a duplicate of ai.stackexchange.com/q/15918/2444. If it's not a duplicate, please, edit your post and clarify why. $\endgroup$
    – nbro
    Nov 1 '20 at 14:13

Browse other questions tagged or ask your own question.