New answers tagged q-learning
0
There are three problems
Limited capacity Neural Network (explained by John)
Non-stationary Target
Non-stationary distribution
Non-stationary Target
In tabular Q-learning, when we update a Q-value, other Q-values in the table don't get affected by this. But in neural networks, one update to the weights aiming to alter one Q-value ends up affecting other Q-...
4
If you're interested in the theory behind Double Q-learning (not deep!), the reference paper would be Double Q-learning by Hado van Hasselt (2010).
As for Double deep Q-learning (also called DDQN, short for Double Deep Q-networks), the reference paper would be Deep Reinforcement Learning with Double Q-learning by Van Hasselt et al. (2016), as pointed out ...
2
You should first read the introductory paper of Double DQN.
https://arxiv.org/abs/1509.06461
Then, depending on what you would like to do, search for other relevant papers that use this method.
4
The way you have described tends to be the common approach. There are of course other ways that you could do this e.g. using an exponential decay, or to only decay after a 'successful' episode, albeit in the latter case I imagine you would want to start with a smaller $\epsilon$ value and then decay by a larger amount.
3
The update form $\theta^{\prime} \leftarrow \tau \theta+(1-\tau) \theta^{\prime}$ (where $\theta'$ and $\theta$ represent the weights of the target network and the current network, respectively) does exist and is correct.
It is called soft update and it has been used in the Deep Deterministic Policy Gradient (DDPG) paper, which uses the concept of a target ...
2
You have two options, either interpolate or restrict the actions only to values that produce states which are in your state vector.
The simplest interpolation scheme is a linear interpolation, which works as follows (assuming DS contains a set of grid points in increasing order). For a state $s'$ you can locate its closest neighbours from the array DS and ...
4
TL;DR: It is Q learning. However Q learning is basically sample-based value iteration, so not surprising you see a similarity.
Q learning* and value iteration are very strongly related. When considering action values, both approaches use the same Bellman equation for optimal policy, $q^*(s,a) = \sum_{r,s'}p(r,s'|s,a)(r+\gamma \text{max}_{a'} q^*(s', a'))$ ...
0
$Q(s,a)$ denotes the $Q-value$ for the state-action pair. It means the expected returns if we start from state $s$, take action $a$, and act according to whatever policy we are currently following.
Suppose we are in state $s_0$, take action $a_0$. To compute the returns, we would need to follow our current policy from whatever state we land up after taking $...
1
Can we make the game faster so that model will be training faster?
It depends on how much processing is required to run the simulation, how efficient that is implemented in whichever library you have loaded, and whether there is anything non-necessary for training that you can disable. Some environments for instance deliberately run "real time" so humans ...
1
The first one is the update rule that we use in the $Q$-learning algorithm.
The second one is the "definition" of $Q(s, a)$ values, although I would personally write it as follows, with an expectation around the reward, to also support cases where rewards might be non-deterministic;
$$Q(s, a) \doteq \mathbb{E} \left[ r(s, a) \right] + \gamma \max_a Q(s', a)...
0
If your intention is to learn make the agent learn which has the min arbitrary value, then you would need to modify your rewards a bit.
The current reward structure provides the incentive to just move to a stage where it gets a reward.
For example, if it is at state 0, it gets the same reward to go to either state 2 or state 3, since both of them have a ...
1
The video you linked is not using reinforcement learning (RL). It is using genetic algorithms (GA).
GA is designed around using multiple agents and picking the best performing to move forward to next generation. With this approach, it is common to want to only view the best performing agents, as the learning mechanism uses the same selection process - the ...
1
Model your problem as an MDP
To solve a problem with reinforcement learning, you need to model your problem as a Markov decision process (MDP), so you need to define
the state space,
the action space, and
the reward function
of the MDP.
Understand your problem and the goal
To do define these, you need to understand your problem and define it as a goal-...
2
First, some preliminary questions: in this case, what is the optimal policy?
It is the policy that maximises return from any given time step $G_t$. You need to be careful with your definition of return with continuing environments. The simple expected sum of future rewards is likely to be positive or negative infinity.
There are three basic approaches:
...
1
In a toy environment, this is a choice you can make relatively freely, depending on what you want to achieve with the learning challenge.
It may help if you think through what the actual consequences for making the "wrong" move are in your environment. There are a few self-consistent options:
The move simply cannot be made and count as playing the game as ...
3
Reinforcement learning (RL) control maximises the expected sum of rewards. If you change the reward metric, it will change what counts as optimal. Your reward functions are not the same, so will in some cases change the priority of solutions.
As a simple example, consider a choice between trajectories with costs A(0,4,4,4) and B(1,1,1,1). In the original ...
0
No, mainly because these are all stochastic approximations and may not represent the true values.
Almost nothing good can be said about NN approximations to value and Q functions(at least according to a professor I have had).
0
Differences between Q-Learning and SARSA:
| SARSA | Q-learning |
|-------|------------|
| Choosing A | π | κ |
| Choosing A' | π | π |
where π is a ε-greedy policy (e.g. ε > 0 with exploration), and κ is a greedy policy (e.g. ε == 0, NO exploration), A is the chosen action from current state s and A' ...
4
If we write the pseudo-code for the SARSA algorithm we first initialise our hyper-parameters etc. and then initialise $S_t$, which we use to choose $A_t$ from our policy $\pi(a|s)$. Then for each $t$ in the episode we do the following:
Take action $A_t$ and observe $R_{t+1}$, $S_{t+1}$
Choose $A_{t+1}$ using $S_{t+1}$ in our policy
$Q(S_t, A_t) = Q(S_t, A_t)...
1
I will try to answer the question in a lesser mathematical (and hopefully correct way).
NOTE: I have used $V_{\pi}$ and $v_{\pi}$ interchangeably.
We start from LHS:
$$\max_s \Bigl\lvert \mathbb{E}_{\pi} \left[ G_{t:t+n} \mid S_t = s \right] - v_{\pi}(s) \Bigr\rvert$$
This can be written in terms of trajectories. Say the probability of observing a $n$ ...
1
I believe that the author is referring to how the networks are trained in Deep RL. Consider Deep Q-Learning where the $Q(s,a)$ is approximated using a neural network. Then the loss function used to train the network is
$$\mathbb{E}[(r + \gamma \max_{a'} Q(s',a') - Q(s,a))^2]\;.$$
Here, $r + \gamma \max_{a'} Q(s',a')$ is your target, what you want your ...
2
It should be possible to train an agent using some variant of DQN to beat a random agent around 100% of the time within a few thousand games.
It may require one or two more advanced techniques to get the learning time down to a low number of thousands. However, if your agent is winning ~50% of games against a random agent, something has gone wrong, since ...
0
The thing is, that while I was posting the question, I tried to tweak with the parameters and it seems that my discount rate (set for 0.99) was causing these errors.
Also - it seems that the is_slippery argument passed to the environment (True - the agent's action will be fulfilled 33% of the times, the rest of the times will be in a random direction, False ...
1
Once you have estimated the $Q$ function, you can derive the policy from it in different ways. For example, you can act greedily with respect to it (see this answer), which can be formally denoted as
$$
\pi(s) = \operatorname{argmax}_{a^*}Q(s, a), \; \forall s \in \mathcal{S}
$$
where $Q(s, a)$ is your estimated value function and $\pi$ the policy greedily ...
0
should I let both 'players' use, and update, the same Q Table?
Yes this works well for zero-sum games, when player 1 wants to maximise a result (often just +1 for "player 1 wins") and player 2 wants to minimise a result (score -1 for "player 2 wins"). That alters algorithms such as Q-learning because the greedy choice switches beween min and max functions ...
Top 50 recent answers are included
