Questions tagged [reinforcement-learning]
For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.
1,773
questions
0
votes
0answers
12 views
Scrabble-MuZero: combine observation planes of different shape
I'm working on an implementation of Scrabble with MuZero. The board state is represented by a matrix with shape $15 \times15 \times 27$ ($26$ letters $+ 1$ wildcard, value $0/1$) and the rack state $...
1
vote
1answer
38 views
In the cross-entropy method, should I select state-action pairs by their immediate reward or by the episode reward?
I am trying to understand the code mechanics when selecting the elite states and elite actions. It appears clear to me that they are those that appear in the episodes with the rewards bigger than the ...
2
votes
0answers
27 views
How to prove Lemma 1.6 in the book “Reinforcement Learning: Theory and Algorithms”
I am trying to prove the following lemma from Reinforcement Learning: Theory and Algorithms on page 8.
Lemma 1.6. We have that:
$$
\left[(1-\gamma)\left(I-\gamma P^{\pi}\right)^{-1}\right]_{(s, a),\...
1
vote
0answers
22 views
Scrabble rack observation with MuZero
Currently I'm trying to implement Scrabble with MuZero.
The $15 \times 15$ game board observation (as input) is of size $27 \times15 \times15$ (26 letters + 1 wildcard) with a value of 0 or 1.
However ...
0
votes
0answers
11 views
Playing Connect Four with reinforcement learning
I'm trying to do self-play reinforcement learning on a board game called Connect Four and I'm not getting good results so would appreciate some ideas on how to improve.
I'm using the PPO algorithm ...
0
votes
1answer
18 views
illegal action masking with Acme DQN agent, dictionary observation space [closed]
I'm trying to add illegal action masking to my dqn agent using masked_epsilon_greedy. Does anyone know how I can update the policy network to use 'observation["your_key_for_observation"]' ...
0
votes
0answers
14 views
KL divergence coefficient update doesn't make sense in RLlib's PPO implementation
I am using RLlib (Ray 1.4.0)'s implementation of PPO for a multi-agent scenario with continuous actions, and I find that the loss includes the KL divergence penalty term, apart from the surrogate loss,...
0
votes
1answer
29 views
How to approach a two-agent two-step action game?
A simple two-player sniper game:
Each player has 9 houses that he can reside in. So 18 houses in total. The houses can be considered in a row: e.g. 1-9 for player A, and 10-18 for player B.
Each ...
1
vote
0answers
12 views
Would the reward normalization be wrong in early episodes?
It's confusing me that how can we normalize the reward without actually knowing the true mean and variance of the reward distribution, specifically, at the early steps and episodes. This may cause ...
5
votes
0answers
42 views
Proof that there always exists a dominating policy in an MDP
I think that it is common knowledge that for any infinite horizon discounted MDP $(S, A, P, r, \gamma)$, there always exists a dominating policy $\pi$, i.e. a policy $\pi$ such that for all policies $\...
0
votes
1answer
44 views
The model learns well, but the validation decreases over time [closed]
I have trained a model for four days. I noticed a behaviour quite strange/unnatural.
During the training, the score and loss look like this:
However, when I see the validation score, I got:
It seems ...
1
vote
0answers
22 views
Doubt in Sutton & Barto's off-policy Monte Carlo control algorithm
The algorithm is described as below:
My understanding: In the third last step, we act greedily w.r.t $Q$. Since we use importance sampling, this $Q \approx Q_\pi$. However, in the next step, whenever ...
0
votes
0answers
31 views
How to normalize rewards in REINFORCE?
I'm trying to solve a reinforcement learning problem using a Monte Carlo policy gradient algorithm and, more specifically, REINFORCE, with rewards attributed to individual moves instead of applied to ...
-1
votes
0answers
19 views
Reinforcement learning agent always takes the same action [closed]
I have trained an RL agent using DQN algorithm. The training was successful. But when I test this trained agent, it is always taking the same action, irrespective of state, this action is the 1st ...
0
votes
0answers
30 views
Defining states and possible actions in Q learning
I am trying to define the number of states and possible actions for a reinforcement learning problem that I want to solve with Q-learning, but I am a bit confused, as I'm totally new to reinforcement ...
1
vote
0answers
16 views
Is there a gentle introduction to reinforcement learning applied to MDPs with continuous state spaces?
I am looking for a gentle introduction (videos, lecture notes, tutorials, books) on reinforcement learning (MDPs) involving continuous states (or very large cardinality of state space). In particular, ...
0
votes
0answers
37 views
Why don't I get the same results of Q-Learning as in Aurélion Géron's Hands-on Machine Learning book?
I noticed something rather intriguing while testing the Deep Q-Network implementation from Aurélion Géron's book Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition; I copy-...
0
votes
0answers
10 views
PPO inputs giving different results in the same environment
I'm using Unity Ml-agents to learn more about reinforcement learning. I've created the most basic environment I could think of, but I'm getting some interesting results.
Take a look at the picture ...
0
votes
1answer
26 views
Where can I find the original conference paper that introduced Q-learning and Deep Q-Learning?
I tried searching a lot, but I could neither find the paper that introduced Q-Learning nor the paper that introduced Deep Q Learning. If anyone knows anything about it please do tell me.
1
vote
0answers
14 views
What is the sample complexity of Monte Carlo Exploring Starts in RL?
We can use a model-free Monte Carlo approach to solving an MDP $(S,A,R,P,\gamma)$ with transition dynamics $P$ unknown by estimating Q-values by rolling out trajectories starting from random states $...
0
votes
1answer
29 views
How to interpret the policy gradient expression in reinforcement learning?
I'm currently going through the OpenAI's spinning up introduction course to reinforcement learning. On one of the final sections, they derive an expression for the gradient of the undiscounted return ...
0
votes
0answers
26 views
tanh activation function output is not between -1 and 1 for continuous action PPO
I am using RLlib's (Ray = 1.4.0) PPO policy, and my first layer after the input (Conv layer) is producing a strange output keeping in mind that the activation for the output is a tanh, which I do ...
0
votes
0answers
21 views
RLLib - What exactly do the avail_action and action_embed_size represent? How do they work with the action_mask to phase out invalid actions?
So, I'm fairly new to reinforcement learning and I needed some help/explanations as to what the action_mask and avail_action fields alongside the action_embed_size actually mean in RLlib (the ...
1
vote
1answer
48 views
Why do I get bad results no matter my neural network function approximator for parametrized Q-learning implementation for Contextual Bandits?
I'd like to ask you why, no matter my neural network function approximator for parametrized Q-learning implementation for a Contextual Bandits environment, I'm getting bad results. I don't know if it'...
1
vote
1answer
23 views
How to define a continuous action distribution with a specific range for Reinforcement Learning?
Specifically for continuous control PPO, let's say my action space range is between $X$ (low) and $Y$ (high) and they are all sampled from a Gaussian Action Distribution with mean $\mu$ and standard ...
3
votes
2answers
188 views
How can we find the value function by solving a system of linear equations?
I am following the book "Reinforcement Learning: An Introduction" by Richard Sutton and Andrew Barto, and they give an example of a problem for which the value function can be computed ...
0
votes
1answer
13 views
TD3 sticking to end values
I am using TD3 to train custom gym environment, but the problem is action values stick to the end.
Sticking to the end values makes reward negative, to be positive it must find action values somewhere ...
0
votes
0answers
16 views
Multi-agent policy gradient, 1 total reward instead of reward in each step, 2 changing action space
I am new in reinforcement learning and not sure I have the right understanding of multi-agent policy gradient.
1, in my question, each agent has its own action space. When doing the sampling, for each ...
0
votes
0answers
27 views
RLlib's Multi-agent PPO continuous actions turn into nan
After some amount of training on a custom Multi-agent sparse-reward environment using RLlib's (1.4.0) PPO network, I found that my continuous actions turn into nan (explodes?) which is probably caused ...
1
vote
3answers
111 views
Can people use neural networks without providing the set of training data?
It seems that neural networks (NNs) can be applied to supervised learning, unsupervised learning and reinforcement learning. Some people even train neural networks without the set of training data. If ...
0
votes
0answers
22 views
Can people set loss function of neural network by themselves instead of choosing cross entropy or mean square error?
I found people used deep neural network to get optimal policy by solving a nonconvex optimization problem. Moreover, they didn't use any set of training data and claimed that it's the difference ...
0
votes
1answer
12 views
In a DDQN architecture, why is the value of a state assumed to be the average of the Q values of the actions?
In a Dueling DQN agent (Wang et al.), the Q function is decomposed as
$$
Q(s, a)=V(s) + A(s, a) - \frac{1}{|A|}\sum_{a'\in \mathcal{A}}A(s, a')
$$
representing the value of the state, plus the ...
1
vote
1answer
53 views
Which policy has to be followed by a player while construction of its own Q-table?
Consider the scenario, where there are two players. One of the players perform the action randomly, whereas I want second player as a Q-player. I mean, the player selects a best action from the Q-...
0
votes
0answers
10 views
Actor-critic reinforcement learning updates and episode length
I am currently using a TD3 agent-critic network to control a vehicle suspension system, where the reward (or rather a penalty) is based on the vertical acceleration of the mass and is calculated at ...
0
votes
1answer
16 views
GLIE MC control (reinforcement learning): how the policy affects evaluation?
In his lecture 5 of the course "Reinforcement Learning", David Silver introduced GLIE Monte-Carlo Control.
I understand that we do policy evaluation for one step and then policy improvement....
0
votes
0answers
18 views
How should we interpret “common coarsening” in this proof of the uniqueness of coarsest bisimulation?
On page 4 of this pdf in a theoretical RL course, we have a proof of the uniquness of the coarsest bisimulation.
A bisimulation $\phi$ is a mapping from states $s \in\mathcal{S}$ to abstract states $\...
0
votes
1answer
58 views
Which policy do I need to use in updating Q function?
Policy function can be of two types: deterministic policy and stochastic policy.
Deterministic policy is of the form $\pi : S \rightarrow A$
Stochastic policy is defined using conditional probability ...
0
votes
0answers
22 views
Can we combine policy evaluation and value iteration steps?
In Sutton & Barto (2nd edition), at the very end on page 83, the following is mentioned:
In general, the entire class of truncated policy iteration algorithms can be thought of as sequences of ...
1
vote
0answers
26 views
Is there a way to beat AlphaGo Zero with different method?
As I read the research from
https://deepmind.com/research
It seem AlphagoZero use zero knowledge and use Reinforcement learning to improve the ai skill of playing.
Is there a way to beat AlphagoZero? ...
1
vote
1answer
62 views
In addition to the reward function, which other functions do I need to implement Q-learning?
In general, $Q$ function is defined as
$$Q : S \times A \rightarrow \mathbb{R}$$
$$Q(s_t,a_t) = Q(s_t,a_t) + \alpha[r_{t+1} + \gamma \max\limits_{a} Q(s_{t+1},a) - Q(s_t,a_t)] $$
$\alpha$ and $\gamma$...
1
vote
1answer
21 views
What do the terms 'Bellman backup' and 'Bellman error' mean?
Some RL literature use terms such as: 'Bellman backup' and 'Bellman error'. What do these terms refer to?
1
vote
1answer
26 views
Doubt regarding policy improvement step in value iteration
I am referring to the Value Iteration (VI) algorithm as mentioned in Sutton's book below.
Rather than getting the greedy deterministic policy after VI converges, what happens if we try to obtain the ...
1
vote
1answer
28 views
How do I represent sample efficiency of RL rewards in mathematical notation?
I define sample efficiency as the area under the curve/graph, where $x$-axis is the number of episodes while y-axis is the cumulative reward for that episode. I would like to formally define it with a ...
1
vote
0answers
19 views
Recommended literature on layers for reinforcement learning
I was recommended to ask here after I posted on stack overflow wrongly.
I was wondering if anyone had any recommended readings on layers used in neural networks for reinforcement learning?
I've been ...
1
vote
1answer
66 views
How does the neural network learn when used in the REINFORCE algorithm?
As per my understanding, you run an entire episode, which contains many steps, and then back-propagate using just 1 loss value. How does the neural network learn to differentiate between good and bad ...
1
vote
1answer
13 views
Is it possible to have values of the states equal to $0$ at the end of the value iteration?
I am new to Reinforcement Learning and I am trying to self learn it. I have already posted some quesiton here and your answershave been really useful to me, so here I am posting another one.
I am ...
1
vote
0answers
14 views
Are there guiding principles as to which activation functions suit a given RL algorithm?
Are there rules of thumb as to which activation functions work well (or which one would not) on the policy and value network of a class of RL algorithms? For hidden layers and for the output layer.
...
1
vote
1answer
33 views
How do we get from conditional expectation on both state and action to only state in the proof of the Policy Improvement Theorem?
I'm going through Sutton and Barto's book Reinforcement Learning: An Introduction and I'm trying to understand the proof of the Policy Improvement Theorem, presented at page 78 of the physical book.
...
1
vote
0answers
29 views
Too slow search using MCTS in OpenAI Atari games
I'm recently using Monte Carlo Tree Search in OpenAi Gym Atari, but the result isn't satisfying.
Without render, the game lasts about 180 steps ( env.step() was called this much time ) with random ...
0
votes
1answer
41 views
How does CURL extract labels from logits?
While going over the pseudocode of the CURL paper, the method to identify labels from the logits wasn't clear to me. I believe this technique might be common in other PyTorch/Deep Learning tasks. I ...
