Linked Questions

8 votes
1 answer
8k views

What is the difference between a stationary and a non-stationary policy?

In reinforcement learning, there are deterministic and non-deterministic (or stochastic) policies, but there are also stationary and non-stationary policies. What is the difference between a ...
nbro's user avatar
  • 40.8k
3 votes
2 answers
1k views

What is a probability distribution in machine learning?

If we were learning or working in the machine learning field, then we frequently come across the term "probability distribution". I know what probability, conditional probability, and ...
Eka's user avatar
  • 1,066
8 votes
2 answers
971 views

What is experience replay in laymen's terms?

I've been reading Google's DeepMind Atari paper and I'm trying to understand the concept of "experience replay". Experience replay comes up in a lot of other reinforcement learning papers (...
user491626's user avatar
5 votes
2 answers
1k views

Given two optimal policies, is an affine combination of them also optimal?

If there are two different optimal policies $\pi_1, \pi_2$ in a reinforcement learning task, will the linear combination (or affine combination) of the two policies $\alpha \pi_1 + \beta \pi_2, \alpha ...
yang liu's user avatar
3 votes
2 answers
1k views

When should one prefer using Total Variational Divergence over KL divergence in RL

In RL, both the KL divergence (DKL) and Total variational divergence (DTV) are used to measure the distance between two policies. I'm most familiar with using DKL as an early stopping metric during ...
mugoh's user avatar
  • 531
1 vote
1 answer
2k views

What kind of problems is DQN algorithm good and bad for?

I know this is a general question, but I'm just looking for intuition. What are the characteristics of problems (in terms of state-space, action-space, environment, or anything else you can think of) ...
mark mark's user avatar
  • 763
3 votes
1 answer
2k views

Why do RL implementations converge on one action?

I have seen this happening in implementations of state-of-the-art RL algorithms where the model converges to a single action over time after multiple training iterations. Are there some general ...
JAYDEEP GHOSE's user avatar
3 votes
1 answer
418 views

Does stochasticity of an environment necessarily mean non-stationarity in MDPs?

Is a stochastic environment necessarily also non-stationary? To elaborate, consider a two-state environment ($s_1$ and $s_2$), with two actions $a_1$ and $a_2$. In $s_1$, taking action $a_1$ has a ...
O'Jhene's user avatar
  • 75
4 votes
2 answers
581 views

What is the difference between return and expected return?

At a time step $t$, for a state $S_{t}$, the return is defined as the discounted cumulative reward from that time step $t$. If an agent is following a policy (which in itself is a probability ...
digi philos's user avatar
4 votes
1 answer
485 views

Is tabular Q-learning considered interpretable?

I am working on a research project in a domain where other related works have always resorted to deep Q-learning. The motivation of my research stems from the fact that the domain has an inherent ...
harshal.c's user avatar
  • 141
0 votes
1 answer
370 views

Which policy do I need to use in updating Q function?

Policy function can be of two types: deterministic policy and stochastic policy. Deterministic policy is of the form $\pi : S \rightarrow A$ Stochastic policy is defined using conditional probability ...
satya's user avatar
  • 187
1 vote
1 answer
335 views

What is the difference between the definition of a stationary policy in reinforcement learning and contextual bandit?

A stationary policy is a function that maps a state to a probability distribution of actions. In a contextual bandit problem, a state itself does not include the history. But in a reinforcement ...
Hunnam 's user avatar
  • 227
0 votes
1 answer
322 views

What is the equation for $\pi_*$ in terms of $q_*(s,a)$?

I am trying to solve the following exercise from Sutton and Barto: Sutton and Barto Exercise 3.27 Give an equation for $\pi_*$ in terms of $q_*(s,a)$ However, I am struggling to do so. I know that $\...
user's user avatar
  • 145
-1 votes
1 answer
353 views

Is my understanding correct regarding the difference between policy and plan?

I am confused regarding the difference between policy and plan in reinforcement learning. According to my understanding, when we calculate the value of state using Bellman equation in deterministic ...
AAA's user avatar
  • 111
0 votes
1 answer
39 views

Why no falling off cliff in SARSA for the example in Sutton-Barto?

Sutton-Barto, page 132: The graph to the right shows the performance of the Sarsa and Qlearning methods with "-greedy action selection, " = 0.1. After an initial transient, Q-learning ...
DSPinfinity's user avatar