quest ions
  • Member for 1 year, 1 month
  • Last seen this week
  • United Kingdom
Intuition behind $1-\gamma$ and $\frac{1}{1-\gamma}$ for calculating discounted future state distribution and discounted reward
Accepted answer
3 votes

Question 1 The taylor expansion of $\frac{1}{1-\gamma}$ at $\gamma= 0$ is as follows $$\frac{1}{1-\gamma} = 1 + \gamma + \gamma^2 + \dots$$ When you multiply by $1-\gamma$ you get $$ 1 = (1-\gamma)(1 +...

View answer
It is mathematically correct to use a Policy Gradient method for 1-step trajectories?
Accepted answer
1 votes

The fundamental idea behind policy gradient is just to maximise the return averaged across all probably trajectories, i.e $$\begin{align} J(\theta) &= E[\sum\limits_{t=1}^{\tau}r(s_t,a_t)]\\ &=...

View answer
What is the target output for updating a Deep Q Network
1 votes

There are a couple ways you can define the architecture of a DQN. The most common way of doing it is by taking in the states and outputting the value function of all possible actions - this leads to a ...

View answer
Is my proof of equation 0.6 in the book "Reinforcement Learning: Theory and Algorithms" correct?
Accepted answer
1 votes

The expectation of a sum is equal to the sum of the expectation this just follows from the linearity property of expectations $$ \begin{aligned} E[\sum_{t} f(s_t,a_t)] &= \sum_{\tau} p(\tau)\left(\...

View answer
Can the law of iterated expectation be used on the inner expectation of the DQN cost function described in the DQN paper
0 votes

Using the related question and simplifying notation $$\begin{align} E_{\mu,\pi}[E_{\mathcal{E}}[s'|s,a]^2] &= \sum_\limits{s,a}\left(\sum_\limits{s'}s'p(s'|s,a)\right)^2p(s,a) \\ &=\sum_\...

View answer
Can stochastic gradient descent be properly used in any sample based learning algorithm in Reinforcement Learning?
-1 votes

As David Ireland has mentioned in his answer, despite correlated sampling, learning algorithms can still converge to the optimal policy/value function. The reason learning algorithms still may produce ...

View answer