4

A typical and practical way to measure the convergence to some solution (so not necessarily the optimal one!) of any numerical iterative algorithm (such as RL algorithms) is to check if the current solution has not changed (much) with respect to the previous one. In your case, the solutions are value functions, so you could check if your algorithm has ...


3

I will first explain briefly to you the difference between supervised learning and reinforcement learning to make sure that you don't have any misunderstandings. In supervised learning you are provided with some data $\{(\textbf{x}_i, y_i)\}_{i=1}^n$ where $\textbf{x}_i$ are the features for data point $i$ and $y_i$ is its true label. Now, the aim of ...


3

However, from the blogs and texts I read, the equations are expressed in terms of V and NOT Q. Why is that? MC and TD are methods for associating value estimates to time step based on experienced gained in later time steps. It does not matter what kind of value estimate is being associated across time, because all value functions are expressing the same ...


2

So why is constant-$\alpha$ being used? This is because control scenarios are inherently non-stationary with respect to value functions. Decaying alpha comes with a risk that improvements to the policy will occur progressively more slowly, because the impact to changing the policy will be learned slowly. From my understanding, in stationary environments, ...


2

Since $A_t$ is already determined (because we are calculating $Q(S_t,A_t)$), I think $\pi(A_t|S_t)$ is definitely 1. But what about $\mu (A_t|S_t)$? Is it 1 or not? You could assign values of 1 to each to get the right answer, but the situation is different. You can see that more clearly in the definition of action value, $q(s,a)$: $$q_{\pi}(s,a) = \mathbb{...


1

Because the value of the terminal state is 0 by definition. There is no further reward to be obtained once you reach the terminal state.


1

I am a novice in Reinforcement Learning and I have been struggling for several monthes about the TD()'s logic. Initially it seemed to me that it was a successfull purely heuristic formula without any theoretical foundation. But nowadays, I understand it simply as a mean's calculation, using the recurrent formula that states that when you a have a mean and a ...


1

It seems that another rather controversial point is about the inclusion of evolutionary algorithms as Reinforcement Learning ones. Sutton & Barto do not. They argue that And also: Other people related with the subject, as the HSE University that offers a course in Coursera, Maxim Lapan , or P. Palanisamy (both Packt's authors) include them into the ...


Only top voted, non community-wiki answers of a minimum length are eligible