13
What you are looking for is called "reinforcement learning".
A reinforcement learning algorithm will try to maximize a reward function. This reward represents how "good" or "bad" an action is in the actual context. For example, in the snake game, your reward will be positive for eating an apple and negative when the snake hits a ...
4
Is a stochastic environment necessarily also non-stationary?
No.
A stochastic environment (i.e. an MDP with a transition model $p(s', r \mid s, a)$) can be stationary (i.e. $p$ does not change over time) or non-stationary ($p$ changes over time). Similarly, a deterministic environment, i.e. the probabilities are $1$ or $0$, can also be either stationary or ...
3
Question 1
The taylor expansion of $\frac{1}{1-\gamma}$ at $\gamma= 0$ is as follows
$$\frac{1}{1-\gamma} = 1 + \gamma + \gamma^2 + \dots$$
When you multiply by $1-\gamma$ you get
$$ 1 = (1-\gamma)(1 + \gamma + \gamma^2 + \dots)$$
Which can be equivalently written as
$$1 = (1-\gamma)\sum_\limits{i=0}^{\infty}\gamma^i$$
Hence we can see that by multiplying ...
3
To obtain guarantees of convergence for Q table values, you need to decay the learning rate, $\alpha$, at a suitable rate. Too fast and convergence will be to inaccurate values. Too slow and convergence never happens.
For sticking with theoretical guarantees, the learning rate decay process should generally follow the rule that $\sum_t \alpha_t = \infty$ but ...
3
Q-learning is guaranteed to converge (in the tabular case) under some mild conditions, one of which is that in the limit we visit each state-action tuple infinitely many times. If your random random policy (i.e. 100% exploration) is guaranteeing this and the other conditions are met (which they probably are) then Q-learning will converge.
The reason that ...
3
Conceptually, in general, how is the context being handled in CB, compared to states in RL?
In terms of its place in the description of Contextual Bandits and Reinforcement Learning, context in CB is an exact analog for state in RL. The framework for RL is a strict generalisation of CB, and can be made similar or the same in a few separate ways:
If the ...
3
The notion of a state in reinforcement learning is (more or less) the same as the notion of a context in contextual bandits. The main difference is that, in reinforcement learning, an action $a_t$ in state $s_t$ not only affects the reward $r_r$ that the agent will get but it will also affect the next state $s_{t+1}$ the agent will end up in, while, in ...
2
Many techniques for the exploration/exploitation dilemma that are inspired by multi-armed bandit problems, such as UCB1, assume that you can explicitly enumerate all state-action pairs; in fact, multi-armed bandit problems usually only have just one "state", and then this requirement turns into only requiring the ability to enumerate actions.
In RL ...
2
Both Belief-MDPs and Bayes-Adaptive MDPs (BAMDPs) are special cases of POMDPs and their state space is augmented with a belief over their unobserved/hidden variables.
In a belief-MDP, the hidden variables can change over the course of an episode. (Eg. Both the position and the uncertainty in the position of the robot can vary during an episode).
In a BAMDP, ...
2
It seems that you are getting confused between the definition of a Q-value and the update rule used to obtain these Q-values.
Remember that to simply obtain an optimal Q-value for a given state-action pair we can evaluate
$$Q(s, a) = r + \gamma \max_{a'} Q(s', a)\;;$$
where $s'$ is the state we transitioned into (note that this only holds when obtaining the ...
1
We need to store the action $a$ as it tells us the action that we took in the state that we are backing up.
Suppose we are in state $s$ and we take action $a$, then we will receive a reward $r$ and next state $s'$. The goal of RL, and in particular DQN (I mention DQN as it is the first algorithm that comes to mind when I think of a replay buffer but it is of ...
1
The goals of experience replay as first proposed by Lin (1992) and more recently applied successfully in the DQN algorithm by Mnih et al. (2013) are to break temporal correlations of updates and to prevent forgetting of experiences that might be useful later on.
To meet these goals, the replay buffer should store tuples required in the learning step.
Most ...
1
It's not really an exhaustive list, but Hutter maintains a small list of problems (click on the bullet point "Universal AI Book" here) related to AIXI (a reinforcement learning agent), some of which have already been solved. The money awards are in the range of 50-500 euros, so they are not as financially important as the Millennium Prize Problems.
1
For anyone wondering, I believe to have found the answer:
Yes, it will be an 8x8 plane where all the entries are the same, the number of moves (or mpves with no progress).
There are two repetitions planes (for each position from the most recent T=8 positions):
a) The first repetition plane will be a plane where all the entries are 1's if the position is ...
1
I'm assuming according to your question that you have a fixed batch, or in other words, there's no possibility of further exploration in your settings. If this assumption is true, you have what's known as Batch/Offline Reinforcement Learning.
First, let's check some aspects about this: in Offline RL once that there's no possibility regarding further ...
1
Meta-Reinforcement Learning can refer to a broad range of ideas. Also, different algorithms are SOTA under different evaluation metrics (sample efficiency, agent performance, adaptation speed on a new task, etc)
Assuming that you are referring to the problem of quickly learning/adapting to a new task by training an agent on a distribution of related tasks, ...
Only top voted, non community-wiki answers of a minimum length are eligible
