2022 Developer Survey is open! Take survey.
3 votes

What should the discount factor for the non-slippery version of the FrozenLake environment be?

After trying to understand what was happening by going through the algorithm on paper (below), I found all values were the same, so greedifying with respect to the value function often just had the ...
user avatar
  • 23.2k
3 votes

Can we also estimate $V_{\pi}$ with SARSA?

What you suggest will work, the main restriction is needing to know $\pi$ fully in order to perform the conversion. If you know that you are going to be estimating $V_{\pi}$ from the start, and have a ...
user avatar
  • 23.2k
3 votes

Is it appropriate to represent 'total failure' as an absorbing state?

In an episodic problem, absorbing states are implemented to make the maths work similarly to continuing tasks. It allows one set of equations to cover two types of MDP (continuing and episodic). For ...
user avatar
  • 23.2k
2 votes

Why is training longer not better in reinforcement learning?

Could it be due to catastrophic forgetting/interference? If once the agent reaches 320K steps it becomes good at the task, it might start to experience only success. This could cause the agent to ...
user avatar
2 votes

Gradient bandit algorithm: is $\bar{R}_t$ average of all rewards or average of rewards corresponding to $A_t$?

Question: is $\bar{R}_t$ average of all rewards or average of rewards corresponding to $A_t$? It is average of all rewards seen so far. Usually a rolling recent average, so it slowly adapts to ...
user avatar
  • 23.2k
2 votes

How to construct a reward function for a "wait and see" problem

In general, the term of art for this problem is "early classification." Early classification of time series has been extensively studied for minimizing class prediction delay in time-...
user avatar
  • 236
2 votes

Given a set of trajectories produced by a fixed policy, what is the the standard approach to estimate Q?

Your trajectories must contain rewards, so I'm assuming you've forgotten them in your original post, i.e., we must have $$\tau_j = (s_0^j, a_0^j, r_1^j, ..., s_{N_j}, a_{N_j}, r_{N_j+1})$$ Given that ...
user avatar
2 votes

What is so special about the Bellman Optimality Principle?

In the context of decision theory (and reinforcement learning, which is the trendier name for this field of research nowadays), the Bellman equations are the most important equations because all ...
user avatar
2 votes

How to handled delayed rewards in contextual bandits

The update rules are not any different. However, if you make many other decisions in the meantime, the timestamps that you are able to run estimate updates for will lag behind the current timestamp. ...
user avatar
  • 23.2k
2 votes
Accepted

What is the total number of actions and rewards count

TL;DR In the DQN paper, each environment was trained for 50 million frames, grouped in fours without overlap, so there were 12.5 million state, action, reward next-state records used. The above direct ...
user avatar
  • 23.2k
2 votes

Is it possible to successively train an RL agent on the same environment with different data

You can mitigate catastrophic forgetting by storing the trajectories generated by the actors during training in a replay buffer. Then, you sample trajectories from that replay buffer. This way, each ...
user avatar
1 vote

How to manage impossible actions?

You could code your agent's policy to never select impossible actions. Your other question implies that you are writing your own behaviour policy function (e.g. you asked about implementing a softmax ...
user avatar
  • 23.2k
1 vote
Accepted

Could Softmax Action Selection be useful to solve an episodic task with more than 100000 possible states and 2000 actions?

Your question contains the answer. Use value function approximation. Tabular methods must compute a value for each state. That becomes unfeasible with large state spaces. Function approximators can ...
user avatar
1 vote

Can you make a Neural Network drunk or high?

To achieve "abrupt" behavior we can try to multiply the network weights by some constant > 1 (10 or 100 for example, depends on wanted degree). If we want sillier behavior - we can ...
user avatar
  • 21
1 vote
Accepted

Is it possible learning convergence is lost in Reinforcement Learning as the state space grows? What about catastrophic forgetting?

With tabular reinforcement learning (RL) methods, then catastrophic forgetting does not come into play, as it is a feature of online learning with approximators such as neural networks. Essentially ...
user avatar
  • 23.2k
1 vote
Accepted

Given a set of trajectories produced by a fixed policy, what is the the standard approach to estimate Q?

What would be the standard approach in this case? Better use TD learning or Monte Carlo? Both should be fine, but they might lead to different estimates, if both these things apply: The amount of ...
user avatar
  • 23.2k
1 vote

Are there any guidelines on how to map the state space to integers in the case tabular RL algorithms?

It's hard to prove a negative, but I don't think there are any special considerations for reinforcement learning (RL) when enumerating and tabulating discrete states and actions. Instead, this is a ...
user avatar
  • 23.2k
1 vote
Accepted

Reinforcement learning algorithms for large problems that are not based on a neural network

There are many state-of-the-art reinforcement learning algorithms for large problems with multidimensional continuous state spaces and actions. All of them rely on some sort of function approximator. ...
user avatar
1 vote

How should I write the reward function to teach the agent the rules of this card game?

I would recommend having the reward as the increase to score caused by adding the card in the chosen location. You could optionally include calculations based on partial rows for pairs, three-of-a-...
user avatar
  • 23.2k
1 vote
Accepted

How is policy iteration capable of improving on a deterministic policy?

These statements are not true for policy iteration and dynamic programming: Since the policy is stochastic and the initial state is the same, we'll always take the same path and evaluate the same ...
user avatar
  • 23.2k

Only top scored, non community-wiki answers of a minimum length are eligible