Questions tagged [reinforcement-learning]
For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.
1,781
questions
1
vote
1answer
19 views
Parallelised my training or the dataset
I have some plans in working with Reinforcement Learning in order to predict the stock price movement. For a stock like TSLA, some training features might be the pivot price values and the set of the ...
-1
votes
0answers
10 views
Spinningup Vanilla Policy Gradient - Where does it use critic network? [closed]
I got a bit into the documentation and git pages of VPG of Spinningup.LINK
One big thing that bugs me - I can't find the place where the actor network uses the critic network evaluation in order to ...
1
vote
0answers
16 views
Reinforcement learning algorithms that deal with noisy state observations
I was recently considering training an agent that perform a task by reinforcement learning. Both the state and actions are continuous, but could be discretized if needed. The problem is that in my ...
1
vote
0answers
24 views
What does it mean there is no rollout in AlphaZero's training?
According to a blog post by DeepMind, AlphaZero doesn't have a real rollout.
AlphaGo Zero does not use "rollouts" - fast, random games used by other Go programs to predict which player will ...
0
votes
0answers
24 views
At what point are MCTS results discarded in AlphaZeros Training?
Regarding the AlphaZero paper, it is not clear to me when the Monte Carlo Tree Search (MCTS) results will be cleaned up.
I assume this has to happen at some point, since mixing results could lead to ...
0
votes
0answers
6 views
Is it better to model a Contextual Multi-Armed Bandit problem as an MDP with a non-zero discount factor than treating it as it is?
I'd like to ask if it is, generally, better to model a problem that naturally appears as a Contextual Multi-Armed Bandit like Recommender Systems as an Markov Decision Process with a non-zero discount ...
1
vote
1answer
26 views
How to use a heuristic policy to increase sample efficiency of a deep reinforcement learning agent?
I have a heuristic solution to a problem which works quite well when certain environmental parameters are known and unchanging. However, in a real world setting these parameters will not be known and ...
1
vote
1answer
24 views
Creating DQN Learning Agent without Gym environment for a custom project
In a project for college I created a simple turn based game, with up to 4 players that can either move or attack the opponents. The players are playing over the network, meaning the clients are ...
0
votes
0answers
23 views
Reinforcement learning applicable to a scheduling problem?
I have a certain scheduling problem and I would like to know in general whether I can use Reinforcement learning (and if so what kind of RL) to solve it. Basically my problem is a mixed-integer linear ...
1
vote
0answers
16 views
Recursive Least squares (RLS) for mini batch
For my application I am considering a learning problem where I simulate a bunch of episodes say '$n$' first, and than carry out the recursive least squares update. Similar to $TD(1)$.
I know that RLS ...
0
votes
0answers
21 views
PPO: decreasing rewards as steps increase
How to explain this? Is this normal? Does this mean a bad design of the environment?
1
vote
0answers
38 views
Why doesn't anyone use reinforcement learning to find the best possible alternative to backpropagation?
To be clear, I'm very uninformed on the topic of alternative learning algorithms to backprop, all my knowledge comes from articles like these:
lets-not-stop-at-backprop
backprop-alternatives
we-need-a-...
0
votes
1answer
23 views
What is the advantage of RL compared with my simple classic algorithm for the MountainCarEnv?
What is the advantage of RL compared with the following simple classic algorithm for the MountainCarEnv? Considering that it takes a long time to train the agent ...
0
votes
0answers
15 views
Scrabble-MuZero: combine observation planes of different shape
I'm working on an implementation of Scrabble with MuZero. The board state is represented by a matrix with shape $15 \times15 \times 27$ ($26$ letters $+ 1$ wildcard, value $0/1$) and the rack state $...
1
vote
1answer
38 views
In the cross-entropy method, should I select state-action pairs by their immediate reward or by the episode reward?
I am trying to understand the code mechanics when selecting the elite states and elite actions. It appears clear to me that they are those that appear in the episodes with the rewards bigger than the ...
2
votes
0answers
37 views
How to prove Lemma 1.6 in the book “Reinforcement Learning: Theory and Algorithms”
I am trying to prove the following lemma from Reinforcement Learning: Theory and Algorithms on page 8.
Lemma 1.6. We have that:
$$
\left[(1-\gamma)\left(I-\gamma P^{\pi}\right)^{-1}\right]_{(s, a),\...
1
vote
0answers
23 views
Scrabble rack observation with MuZero
Currently I'm trying to implement Scrabble with MuZero.
The $15 \times 15$ game board observation (as input) is of size $27 \times15 \times15$ (26 letters + 1 wildcard) with a value of 0 or 1.
However ...
0
votes
0answers
13 views
Playing Connect Four with reinforcement learning
I'm trying to do self-play reinforcement learning on a board game called Connect Four and I'm not getting good results so would appreciate some ideas on how to improve.
I'm using the PPO algorithm ...
0
votes
0answers
14 views
KL divergence coefficient update doesn't make sense in RLlib's PPO implementation
I am using RLlib (Ray 1.4.0)'s implementation of PPO for a multi-agent scenario with continuous actions, and I find that the loss includes the KL divergence penalty term, apart from the surrogate loss,...
0
votes
1answer
31 views
How to approach a two-agent two-step action game?
A simple two-player sniper game:
Each player has 9 houses that he can reside in. So 18 houses in total. The houses can be considered in a row: e.g. 1-9 for player A, and 10-18 for player B.
Each ...
1
vote
0answers
14 views
Would the reward normalization be wrong in early episodes?
It's confusing me that how can we normalize the reward without actually knowing the true mean and variance of the reward distribution, specifically, at the early steps and episodes. This may cause ...
5
votes
0answers
48 views
Proof that there always exists a dominating policy in an MDP
I think that it is common knowledge that for any infinite horizon discounted MDP $(S, A, P, r, \gamma)$, there always exists a dominating policy $\pi$, i.e. a policy $\pi$ such that for all policies $\...
0
votes
1answer
44 views
The model learns well, but the validation decreases over time [closed]
I have trained a model for four days. I noticed a behaviour quite strange/unnatural.
During the training, the score and loss look like this:
However, when I see the validation score, I got:
It seems ...
1
vote
0answers
22 views
Doubt in Sutton & Barto's off-policy Monte Carlo control algorithm
The algorithm is described as below:
My understanding: In the third last step, we act greedily w.r.t $Q$. Since we use importance sampling, this $Q \approx Q_\pi$. However, in the next step, whenever ...
0
votes
0answers
34 views
How to normalize rewards in REINFORCE?
I'm trying to solve a reinforcement learning problem using a Monte Carlo policy gradient algorithm and, more specifically, REINFORCE, with rewards attributed to individual moves instead of applied to ...
0
votes
0answers
31 views
Defining states and possible actions in Q learning
I am trying to define the number of states and possible actions for a reinforcement learning problem that I want to solve with Q-learning, but I am a bit confused, as I'm totally new to reinforcement ...
1
vote
0answers
17 views
Is there a gentle introduction to reinforcement learning applied to MDPs with continuous state spaces?
I am looking for a gentle introduction (videos, lecture notes, tutorials, books) on reinforcement learning (MDPs) involving continuous states (or very large cardinality of state space). In particular, ...
0
votes
0answers
37 views
Why don't I get the same results of Q-Learning as in Aurélion Géron's Hands-on Machine Learning book?
I noticed something rather intriguing while testing the Deep Q-Network implementation from Aurélion Géron's book Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition; I copy-...
0
votes
0answers
11 views
PPO inputs giving different results in the same environment
I'm using Unity Ml-agents to learn more about reinforcement learning. I've created the most basic environment I could think of, but I'm getting some interesting results.
Take a look at the picture ...
0
votes
1answer
26 views
Where can I find the original conference paper that introduced Q-learning and Deep Q-Learning?
I tried searching a lot, but I could neither find the paper that introduced Q-Learning nor the paper that introduced Deep Q Learning. If anyone knows anything about it please do tell me.
1
vote
0answers
14 views
What is the sample complexity of Monte Carlo Exploring Starts in RL?
We can use a model-free Monte Carlo approach to solving an MDP $(S,A,R,P,\gamma)$ with transition dynamics $P$ unknown by estimating Q-values by rolling out trajectories starting from random states $...
0
votes
1answer
29 views
How to interpret the policy gradient expression in reinforcement learning?
I'm currently going through the OpenAI's spinning up introduction course to reinforcement learning. On one of the final sections, they derive an expression for the gradient of the undiscounted return ...
0
votes
0answers
26 views
tanh activation function output is not between -1 and 1 for continuous action PPO
I am using RLlib's (Ray = 1.4.0) PPO policy, and my first layer after the input (Conv layer) is producing a strange output keeping in mind that the activation for the output is a tanh, which I do ...
0
votes
0answers
21 views
RLLib - What exactly do the avail_action and action_embed_size represent? How do they work with the action_mask to phase out invalid actions?
So, I'm fairly new to reinforcement learning and I needed some help/explanations as to what the action_mask and avail_action fields alongside the action_embed_size actually mean in RLlib (the ...
1
vote
1answer
52 views
Why do I get bad results no matter my neural network function approximator for parametrized Q-learning implementation for Contextual Bandits?
I'd like to ask you why, no matter my neural network function approximator for parametrized Q-learning implementation for a Contextual Bandits environment, I'm getting bad results. I don't know if it'...
1
vote
1answer
24 views
How to define a continuous action distribution with a specific range for Reinforcement Learning?
Specifically for continuous control PPO, let's say my action space range is between $X$ (low) and $Y$ (high) and they are all sampled from a Gaussian Action Distribution with mean $\mu$ and standard ...
3
votes
2answers
193 views
How can we find the value function by solving a system of linear equations?
I am following the book "Reinforcement Learning: An Introduction" by Richard Sutton and Andrew Barto, and they give an example of a problem for which the value function can be computed ...
0
votes
1answer
13 views
TD3 sticking to end values
I am using TD3 to train custom gym environment, but the problem is action values stick to the end.
Sticking to the end values makes reward negative, to be positive it must find action values somewhere ...
0
votes
0answers
16 views
Multi-agent policy gradient, 1 total reward instead of reward in each step, 2 changing action space
I am new in reinforcement learning and not sure I have the right understanding of multi-agent policy gradient.
1, in my question, each agent has its own action space. When doing the sampling, for each ...
0
votes
0answers
34 views
RLlib's Multi-agent PPO continuous actions turn into nan
After some amount of training on a custom Multi-agent sparse-reward environment using RLlib's (1.4.0) PPO network, I found that my continuous actions turn into nan (explodes?) which is probably caused ...
1
vote
3answers
118 views
Can people use neural networks without providing the set of training data?
It seems that neural networks (NNs) can be applied to supervised learning, unsupervised learning and reinforcement learning. Some people even train neural networks without the set of training data. If ...
0
votes
0answers
22 views
Can people set loss function of neural network by themselves instead of choosing cross entropy or mean square error?
I found people used deep neural network to get optimal policy by solving a nonconvex optimization problem. Moreover, they didn't use any set of training data and claimed that it's the difference ...
0
votes
1answer
12 views
In a DDQN architecture, why is the value of a state assumed to be the average of the Q values of the actions?
In a Dueling DQN agent (Wang et al.), the Q function is decomposed as
$$
Q(s, a)=V(s) + A(s, a) - \frac{1}{|A|}\sum_{a'\in \mathcal{A}}A(s, a')
$$
representing the value of the state, plus the ...
1
vote
1answer
53 views
Which policy has to be followed by a player while construction of its own Q-table?
Consider the scenario, where there are two players. One of the players perform the action randomly, whereas I want second player as a Q-player. I mean, the player selects a best action from the Q-...
0
votes
0answers
10 views
Actor-critic reinforcement learning updates and episode length
I am currently using a TD3 agent-critic network to control a vehicle suspension system, where the reward (or rather a penalty) is based on the vertical acceleration of the mass and is calculated at ...
0
votes
1answer
16 views
GLIE MC control (reinforcement learning): how the policy affects evaluation?
In his lecture 5 of the course "Reinforcement Learning", David Silver introduced GLIE Monte-Carlo Control.
I understand that we do policy evaluation for one step and then policy improvement....
0
votes
0answers
19 views
How should we interpret “common coarsening” in this proof of the uniqueness of coarsest bisimulation?
On page 4 of this pdf in a theoretical RL course, we have a proof of the uniquness of the coarsest bisimulation.
A bisimulation $\phi$ is a mapping from states $s \in\mathcal{S}$ to abstract states $\...
0
votes
1answer
58 views
Which policy do I need to use in updating Q function?
Policy function can be of two types: deterministic policy and stochastic policy.
Deterministic policy is of the form $\pi : S \rightarrow A$
Stochastic policy is defined using conditional probability ...
0
votes
0answers
22 views
Can we combine policy evaluation and value iteration steps?
In Sutton & Barto (2nd edition), at the very end on page 83, the following is mentioned:
In general, the entire class of truncated policy iteration algorithms can be thought of as sequences of ...
1
vote
0answers
27 views
Is there a way to beat AlphaGo Zero with different method?
As I read the research from
https://deepmind.com/research
It seem AlphagoZero use zero knowledge and use Reinforcement learning to improve the ai skill of playing.
Is there a way to beat AlphagoZero? ...
