3
votes
What should the discount factor for the non-slippery version of the FrozenLake environment be?
After trying to understand what was happening by going through the algorithm on paper (below), I found all values were the same, so greedifying with respect to the value function often just had the ...
- 23.2k
3
votes
Can we also estimate $V_{\pi}$ with SARSA?
What you suggest will work, the main restriction is needing to know $\pi$ fully in order to perform the conversion.
If you know that you are going to be estimating $V_{\pi}$ from the start, and have a ...
- 23.2k
3
votes
Is it appropriate to represent 'total failure' as an absorbing state?
In an episodic problem, absorbing states are implemented to make the maths work similarly to continuing tasks. It allows one set of equations to cover two types of MDP (continuing and episodic). For ...
- 23.2k
2
votes
Why is training longer not better in reinforcement learning?
Could it be due to catastrophic forgetting/interference? If once the agent reaches 320K steps it becomes good at the task, it might start to experience only success. This could cause the agent to ...
- 21
2
votes
Gradient bandit algorithm: is $\bar{R}_t$ average of all rewards or average of rewards corresponding to $A_t$?
Question: is $\bar{R}_t$ average of all rewards or average of rewards corresponding to $A_t$?
It is average of all rewards seen so far. Usually a rolling recent average, so it slowly adapts to ...
- 23.2k
2
votes
How to construct a reward function for a "wait and see" problem
In general, the term of art for this problem is "early classification."
Early classification of time series has been extensively studied for minimizing class prediction delay in time-...
- 236
2
votes
Given a set of trajectories produced by a fixed policy, what is the the standard approach to estimate Q?
Your trajectories must contain rewards, so I'm assuming you've forgotten them in your original post, i.e., we must have $$\tau_j = (s_0^j, a_0^j, r_1^j, ..., s_{N_j}, a_{N_j}, r_{N_j+1})$$
Given that ...
2
votes
What is so special about the Bellman Optimality Principle?
In the context of decision theory (and reinforcement learning, which is the trendier name for this field of research nowadays), the Bellman equations are the most important equations because all ...
2
votes
How to handled delayed rewards in contextual bandits
The update rules are not any different.
However, if you make many other decisions in the meantime, the timestamps that you are able to run estimate updates for will lag behind the current timestamp.
...
- 23.2k
2
votes
Accepted
What is the total number of actions and rewards count
TL;DR In the DQN paper, each environment was trained for 50 million frames, grouped in fours without overlap, so there were 12.5 million state, action, reward next-state records used.
The above direct ...
- 23.2k
2
votes
Is it possible to successively train an RL agent on the same environment with different data
You can mitigate catastrophic forgetting by storing the trajectories generated by the actors during training in a replay buffer. Then, you sample trajectories from that replay buffer. This way, each ...
1
vote
How to manage impossible actions?
You could code your agent's policy to never select impossible actions.
Your other question implies that you are writing your own behaviour policy function (e.g. you asked about implementing a softmax ...
- 23.2k
1
vote
Accepted
Could Softmax Action Selection be useful to solve an episodic task with more than 100000 possible states and 2000 actions?
Your question contains the answer. Use value function approximation. Tabular methods must compute a value for each state. That becomes unfeasible with large state spaces. Function approximators can ...
- 911
1
vote
Can you make a Neural Network drunk or high?
To achieve "abrupt" behavior we can try to multiply the network weights by some constant > 1 (10 or 100 for example, depends on wanted degree).
If we want sillier behavior - we can ...
- 21
1
vote
Accepted
Is it possible learning convergence is lost in Reinforcement Learning as the state space grows? What about catastrophic forgetting?
With tabular reinforcement learning (RL) methods, then catastrophic forgetting does not come into play, as it is a feature of online learning with approximators such as neural networks. Essentially ...
- 23.2k
1
vote
Accepted
Given a set of trajectories produced by a fixed policy, what is the the standard approach to estimate Q?
What would be the standard approach in this case? Better use TD learning or Monte Carlo?
Both should be fine, but they might lead to different estimates, if both these things apply:
The amount of ...
- 23.2k
1
vote
Are there any guidelines on how to map the state space to integers in the case tabular RL algorithms?
It's hard to prove a negative, but I don't think there are any special considerations for reinforcement learning (RL) when enumerating and tabulating discrete states and actions. Instead, this is a ...
- 23.2k
1
vote
Accepted
Reinforcement learning algorithms for large problems that are not based on a neural network
There are many state-of-the-art reinforcement learning algorithms for large problems with multidimensional continuous state spaces and actions. All of them rely on some sort of function approximator.
...
- 911
1
vote
How should I write the reward function to teach the agent the rules of this card game?
I would recommend having the reward as the increase to score caused by adding the card in the chosen location. You could optionally include calculations based on partial rows for pairs, three-of-a-...
- 23.2k
1
vote
Accepted
How is policy iteration capable of improving on a deterministic policy?
These statements are not true for policy iteration and dynamic programming:
Since the policy is stochastic and the initial state is the same, we'll always take the same path and evaluate the same ...
- 23.2k
Only top scored, non community-wiki answers of a minimum length are eligible