New answers tagged reinforcement-learning
0
votes
Use of virtual worlds (e.g. Second Life) for training Artificial General Intelligence agents?
Yes, there are actually quite a few examples of this:
https://www.deepmind.com/publications/open-ended-learning-leads-to-generally-capable-agents
https://ieeexplore.ieee.org/abstract/document/9089532/
...
- 1,145
0
votes
What is the advantage of using experience replay (as opposed to feeding it sequential data)?
Random sampling from the replay memory helps us reach closer to the i.i.d assumption and also lets the learning algorithm train on the data multiple times compared to just once. Therefore, there is a ...
- 275
0
votes
Can off-policy algorithms benefit from the parallelization?
From the point of view of someone developing an in-house DRL lib and working on extremely CPU-intensive environments (usually large finite element-based simulations that can require several hours to ...
- 101
0
votes
PPO vs SAC for discrete action spaces
First, both SAC and PPO are usable for continuous and discrete action spaces. However, in the case of discrete action spaces, SAC cost functions must be previously adapted. As explained in this Stable ...
- 1
2
votes
Accepted
$E_{\pi}[R_{t+1}|S_t=s,A_t=a] = E[R_{t+1}|S_t=s,A_t=a]$?
Question: Can I write it without the subscript? So $$E_{\pi}[R_{t+1}|S_t=s,A_t=a] = E[R_{t+1}|S_t=s,A_t=a]$$
Yes, your reasoning is sound, there is no need to condition the expectation on the policy, ...
- 23.8k
4
votes
Accepted
Is there a standardized method to train a reinforcement learning NN by demonstration?
Yes, this is known as imitation learning, which can be divided into
inverse RL (i.e. learn a reward function from demonstrations, then apply RL), and
behaviour cloning (supervised learning applied to ...
- 33.8k
0
votes
In Alpha(Go)Zero, why is the policy extracted from MCTS better than the network one?
Basically, the policy network just looks at the current state and doesn't have any added benefit from searching. I think of the policy network as a chess player's initial candidate move selection and ...
- 121
3
votes
Accepted
Why and when do we need to normalize weights in Reinforcement Learning?
The kind of divergence that the other question experienced is a common problem with deep RL and temporal difference methods (Q-learning, SARSA, or any Actor Critic).
The weight normalisation would not ...
- 23.8k
0
votes
How many training steps does it usually take to train an RL model?
It depends on the the problem you're applying PPO to. To get an idea, you can have a look at the CleanRL benchmarks, there are a few of them where they use PPO: https://wandb.ai/openrlbenchmark/...
- 1
2
votes
Accepted
How do I create an AI controller for Pacman?
About the environments
For the controller part of your question, I would advice looking at openAI gym.
https://www.gymlibrary.ml/content/environment_creation/ #how to make your own gym enviroment
...
- 349
1
vote
How many training steps does it usually take to train an RL model?
This is not possible to know in advance precisely, only approximately, but it also strongly depends on the environment, hyperparameters and algorithm. For hard environments, e.g. the ones learning ...
- 351
1
vote
How to form 10 and 20 actions in corridor environment, in the paper "Dueling Network Architectures for Deep Reinforcement Learning"?
I think you are right. But then the next doubt would be why to include so many no-ops instead of just original action space+ one no-op space rather. And for this, I think it is to intentionally ...
1
vote
Accepted
How to compare memory requirements for tabular Q-learning vs deep neural network?
You don't say, but I suspect from your description, that you have designed the neural network to operate over one-hot-encoding representations of states and actions. Using such a representation offers ...
- 23.8k
1
vote
Is the described Q-table considered large?
There a couple of "rules of thumb" you might apply to decide whether a Q table is large enough that some kind of approximation would help:
Does it fit into memory?
Does the rate of ...
- 23.8k
3
votes
Accepted
What can we learn from AlphaZero in the development towards AGI?
Some learnings from AlphaZero:
Self-play, and more generally sandbox training, is effective. This indicates that given the right enviroment and enough computational power we can build highly ...
- 351
0
votes
Should PPO always converge toward the global optimum?
Even if you were reaching the global optimum for the PPO loss, it wouldn't mean that the learnt policy would resemble the human stride. Nothing in the loss enforces that.
More generally in RL, the ...
1
vote
Replay buffer action range in DDPG
It has an obvious answer: Network is conditioned to use tanh activation. Hence the buffer should be [-1, 1], or unscaled values before action execution. As I am not using openai gym or other baselines ...
- 13
0
votes
Accepted
How to tune hypeparametes in A2C-ppo?
Human performance on Breakout is ~30, if you refer to the original DQN paper (Table 1). In the original A3C paper, it takes around 5 epochs to reach that score, so 20 millions frames (Figure 3).
Is ...
0
votes
Accepted
Should PPO always converge toward the global optimum?
In the world of theory, Convergence Proof for Actor-Critic Methods Applied to PPO and RUDDER claims to show that PPO eventually converges to a local (but not necessarily global) optimum.
In practice, ...
- 451
0
votes
How do we estimate the value of a stochastic policy?
We are choosing actions randomly with probabilities given by the policy $\pi$. For example, one policy might make two actions A and B equally likely, another might choose A 90% of the time.
What we're ...
- 451
2
votes
Why is a large replay buffer inefficient?
I read the same thing recently, and my interpretation was this:
If you only use the very-most recent data, you will overfit to that and things will break
We'd like to train the network to predict ...
- 451
2
votes
Do I need to normalize all state-space variables? If so, how?
I'll start with the literal question in the title:
Do I need to normalize all state-space variables?
You don't strictly need to in theory. It's often really useful, or sometimes borderline necessary,...
- 9,379
0
votes
PPO: multiple discrete actions per step, one depends on the other
What you describe will be possible, but you need to ensure that your policy returns the probability of the action(s) taken.
In the two step process you describe, you would have something like
\begin{...
- 876
2
votes
Accepted
Do I need to normalize all state-space variables? If so, how?
The way I've seen most codes treat the state normalization is that they simply take a running mean and standard deviation for each dimension of the state space. As you point out, this normalization ...
- 876
2
votes
Do I need to normalize all state-space variables? If so, how?
It's likely to train as long as they're reasonably within the orders of magnitude of other normalized variables. The network can adjust for that.
But it might cause problems later, if the values move ...
- 451
0
votes
Accepted
Is it possible to add states to the Q-table after the game has started?
Yes, it is possible to add states to a Q table after the game has started, for example by storing the "table" in a binary tree.
Nonetheless, with a simple interpretation of state (like all ...
- 451
2
votes
Accepted
Does it make sense to provide a DQN with negative rewards for a network with relu and sigmoid activations?
A network with ReLU activation can predict negative values; we put ReLU between the hidden layers but return the output of the final layer without any activation function, or with a linear activation ...
- 451
1
vote
Accepted
How to deal with changing rewards in Q-learning? DQN?
Is my definition of 'state' or 'action' wrong?
I hesitate to say 'wrong', but that's not how state and action are defined in RL, and that mismatch might make the algorithms hard to understand.
In RL ...
- 451
2
votes
Why is exploitation necessary during training?
If you explore too much, you waste your time (among other resources.) You will probably exhaust your resources before you learn anything meaningful.
Let's say your goal is to learn as much about Star ...
- 121
0
votes
Which RL algorithm should I use to learn an optimal weight vector?
My friend propose local random search:
Initialize:
$W_0 = (0.5,...,0.5)$
step = 0.1, $K < N$, MAX_ITERATIONS
Each iteration:
$W_{new}$ <- $W$
randomly choose up to $K$ weights, change each ...
- 365
2
votes
Why is exploitation necessary during training?
There is an additional factor to consider about exploration/exploitation trade-off, that sometimes applies in addition to the reason in the accepted answer and most other answers here.
Sometimes an ...
- 23.8k
2
votes
Why is exploitation necessary during training?
Imagine trying to navigate a maze from the outside. Let's say you lose if you get to a dead end, and win if you get to the middle. After some experience by random trials, we know where some dead ends ...
- 21
8
votes
Accepted
Why is exploitation necessary during training?
An algorithm that chooses to always explore during training is unlikely to find an optimal policy because it will be employing a more random search as opposed to a directed search. During training, ...
- 928
5
votes
Why is exploitation necessary during training?
Exploitation is important during training to help the network encounter and learn to handle situations that don't occur until the network has successfully navigated other situations.
For example, ...
- 451
7
votes
Accepted
Mathematically, what is happening differently in the neural net during exploration vs. exploitation?
Typically, the NN is trained the same way whether an action is chosen for exploration or exploitation. Look at the objective (AKA loss) function for any algorithm you're interested in and you'll ...
- 451
0
votes
Why is exploitation necessary during training?
In supervised ML there is no exploration and exploitation.
In reinforcement learning, the agent in each step has many choices.
So the agent can exploit, meaning gaining the highest reward known to him ...
- 11
3
votes
Is there a way to easily simulate video games, without actually rendering the pixels on screen?
Bypassing graphics
As mentioned in Neil Slater's answer, manipulating the engine to bypass graphics rendering to speed up AI simulation can be a valid approach. I have done that myself.
But there are ...
- 31
4
votes
Accepted
In Value Iteration, why can we initialize the value function arbitrarily?
If the value function of a state $v(s)$ is relatively high, then you are absolutely correct in saying that a greedy policy may choose to visit $s$, since the high $v(s)$ makes it very promising. The ...
- 928
2
votes
Accepted
For continuing tasks, is the choice of episode length completely arbitrary?
There are things that impact ideal pseudo-episode length for learning continuing (non-episodic) environments:
Start state. The start state of a continuing environment may be special in some way and ...
- 23.8k
6
votes
In Value Iteration, why can we initialize the value function arbitrarily?
Is this something to do with the Bellman optimality constraint itself?
That is part of it, and important for episodic problems without discounting. The Bellman equations link between time steps, ...
- 23.8k
3
votes
Accepted
Why are only neural networks (and not SVMs, for example) used for reinforcement learning?
The biggest problem with SVMs, random forests, gradient boosting and others for reinforcement learning (RL) is that they are not able to learn online, adjusting for new data as it arrives, and equally ...
- 23.8k
7
votes
Is there a way to easily simulate video games, without actually rendering the pixels on screen?
It depends on the game environment and on the model being trained.
If you are training an agent that uses vision to decide action, then typically you need a copy of the rendered screen:
If that ...
- 23.8k
1
vote
Accepted
Why aren't neural networks contractions?
why neural networks aren't considered contractions, as Geoffrey J. Gordon says in his paper.
I am not sure how strongly you mean aren't considered, if you mean it in a strong sense or weak sense.
...
- 81
1
vote
Accepted
Why does the average-reward estimator for continuing tasks use the TD error?
Mystery solved thanks to Exercise 10.8 in the book. The reason is that we want the running mean to converge to the actual value of the average reward.
With $\bar{R}_{t + 1} = \bar{R}_t + \beta \delta$...
- 66
Top 50 recent answers are included