Questions tagged [reinforcement-learning]

For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.

Filter by
Sorted by
Tagged with
1
vote
1answer
26 views

Is there any inherent assumption of start and goal states in an MDP?

MDP stands for the Markov decision process. It is a 5-length tuple used in reinforcement learning. $$MDP = (S, A, T, R, \pi)$$ $S$ stands for a set of states, also called state space. $A$ stands for a ...
0
votes
1answer
88 views

Minimizing trial and error in reinforcement learning

The initial environment state is 0.25. Each time step the agent performs a discrete action of 0 or ...
0
votes
1answer
61 views

Deep Q-Learning with multiple discrete actions

I am working on a DQN project with Pytorch, where I should choose multiple discrete actions, each in a range, say, (0, 15). I am wondering how I can model it, such ...
2
votes
0answers
33 views

How do neural networks deal with inputs of different sizes that are padded in order to have them of the same size?

I am trying to create an environment for RL where the size of my input (observation space) is not fixed. As a way around it, I thought about padding the size to a maximum value and then assigning &...
0
votes
0answers
16 views

Reinforcement learning: better performance with a stochastic environment

I modified the cartpole environment with one line to make it stochastic: ...
-1
votes
1answer
50 views

Reinforcement learning environment design for crypto trading [closed]

how should one design the observation and reward for a crypto trading environment? suppose the ...
0
votes
0answers
28 views

Similarities Between Evolutionary Algorithms and Reinforcement Learning [duplicate]

Evolutionary algorithms use the fitness function to score agents and tend to choose the one with the high score. This tends to maximize the score of surviving agents. Doesn't reinforcement learning ...
2
votes
1answer
53 views

If we can model the environment, wouldn't be meaningless to use a model-free algorithm?

I am trying to understand the concept of model-free and model-based approaches. As far as I understand, having a model of the environment does not mean that an RL agent has to be model-based. It is ...
0
votes
0answers
40 views

Can reinforcement learning be used to learn an unknown analytical function (for example, $y = x^2$ )?

Are there any examples for RL to learn analytical functions (for example, $y=x^2$)? What are the considerations when constructing the environment? Are there any literature that analyzes the difficulty/...
1
vote
1answer
224 views

Parallelised my training or the dataset

I have some plans in working with Reinforcement Learning in order to predict the stock price movement. For a stock like TSLA, some training features might be the pivot price values and the set of the ...
1
vote
0answers
30 views

Reinforcement learning algorithms that deal with noisy state observations

I was recently considering training an agent that perform a task by reinforcement learning. Both the state and actions are continuous, but could be discretized if needed. The problem is that in my ...
2
votes
1answer
104 views

What does it mean there is no rollout in AlphaZero's training?

According to a blog post by DeepMind, AlphaZero doesn't have a real rollout. AlphaGo Zero does not use "rollouts" - fast, random games used by other Go programs to predict which player will ...
0
votes
0answers
87 views

At what point are MCTS results discarded in AlphaZero Training?

Regarding the AlphaZero paper, it is not clear to me when the Monte Carlo Tree Search (MCTS) results will be cleaned up. I assume this has to happen at some point, since mixing results could lead to ...
0
votes
0answers
11 views

Is it better to model a Contextual Multi-Armed Bandit problem as an MDP with a non-zero discount factor than treating it as it is?

I'd like to ask if it is, generally, better to model a problem that naturally appears as a Contextual Multi-Armed Bandit like Recommender Systems as an Markov Decision Process with a non-zero discount ...
1
vote
1answer
44 views

How to use a heuristic policy to increase sample efficiency of a deep reinforcement learning agent?

I have a heuristic solution to a problem which works quite well when certain environmental parameters are known and unchanging. However, in a real world setting these parameters will not be known and ...
1
vote
1answer
46 views

Creating DQN Learning Agent without Gym environment for a custom project

In a project for college I created a simple turn based game, with up to 4 players that can either move or attack the opponents. The players are playing over the network, meaning the clients are ...
2
votes
1answer
110 views

Reinforcement learning applicable to a scheduling problem?

I have a certain scheduling problem and I would like to know in general whether I can use Reinforcement learning (and if so what kind of RL) to solve it. Basically my problem is a mixed-integer linear ...
1
vote
0answers
16 views

Recursive Least squares (RLS) for mini batch

For my application I am considering a learning problem where I simulate a bunch of episodes say '$n$' first, and than carry out the recursive least squares update. Similar to $TD(1)$. I know that RLS ...
2
votes
1answer
170 views

What do equations 1 and 3 describe in the "Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels" paper?

This paper uses image augmentation to improve RL algorithms. It contains the following paragraph - "Our approach, DrQ, is the union of the three separate regularization mechanisms introduced ...
0
votes
0answers
25 views

PPO: decreasing rewards as steps increase

How to explain this? Is this normal? Does this mean a bad design of the environment?
1
vote
0answers
41 views

Why doesn't anyone use reinforcement learning to find the best possible alternative to backpropagation?

To be clear, I'm very uninformed on the topic of alternative learning algorithms to backprop, all my knowledge comes from articles like these: lets-not-stop-at-backprop backprop-alternatives we-need-a-...
0
votes
1answer
27 views

What is the advantage of RL compared with my simple classic algorithm for the MountainCarEnv?

What is the advantage of RL compared with the following simple classic algorithm for the MountainCarEnv? Considering that it takes a long time to train the agent ...
0
votes
0answers
15 views

Scrabble-MuZero: combine observation planes of different shape

I'm working on an implementation of Scrabble with MuZero. The board state is represented by a matrix with shape $15 \times15 \times 27$ ($26$ letters $+ 1$ wildcard, value $0/1$) and the rack state $...
1
vote
1answer
39 views

In the cross-entropy method, should I select state-action pairs by their immediate reward or by the episode reward?

I am trying to understand the code mechanics when selecting the elite states and elite actions. It appears clear to me that they are those that appear in the episodes with the rewards bigger than the ...
2
votes
0answers
42 views

How to prove Lemma 1.6 in the book "Reinforcement Learning: Theory and Algorithms"

I am trying to prove the following lemma from Reinforcement Learning: Theory and Algorithms on page 8. Lemma 1.6. We have that: $$ \left[(1-\gamma)\left(I-\gamma P^{\pi}\right)^{-1}\right]_{(s, a),\...
1
vote
0answers
24 views

Scrabble rack observation with MuZero

Currently I'm trying to implement Scrabble with MuZero. The $15 \times 15$ game board observation (as input) is of size $27 \times15 \times15$ (26 letters + 1 wildcard) with a value of 0 or 1. However ...
0
votes
0answers
16 views

Playing Connect Four with reinforcement learning

I'm trying to do self-play reinforcement learning on a board game called Connect Four and I'm not getting good results so would appreciate some ideas on how to improve. I'm using the PPO algorithm ...
0
votes
0answers
33 views

KL divergence coefficient update doesn't make sense in RLlib's PPO implementation

I am using RLlib (Ray 1.4.0)'s implementation of PPO for a multi-agent scenario with continuous actions, and I find that the loss includes the KL divergence penalty term, apart from the surrogate loss,...
0
votes
1answer
32 views

How to approach a two-agent two-step action game?

A simple two-player sniper game: Each player has 9 houses that he can reside in. So 18 houses in total. The houses can be considered in a row: e.g. 1-9 for player A, and 10-18 for player B. Each ...
1
vote
0answers
21 views

Would the reward normalization be wrong in early episodes?

It's confusing me that how can we normalize the reward without actually knowing the true mean and variance of the reward distribution, specifically, at the early steps and episodes. This may cause ...
5
votes
0answers
50 views

Proof that there always exists a dominating policy in an MDP

I think that it is common knowledge that for any infinite horizon discounted MDP $(S, A, P, r, \gamma)$, there always exists a dominating policy $\pi$, i.e. a policy $\pi$ such that for all policies $\...
0
votes
1answer
44 views

The model learns well, but the validation decreases over time [closed]

I have trained a model for four days. I noticed a behaviour quite strange/unnatural. During the training, the score and loss look like this: However, when I see the validation score, I got: It seems ...
2
votes
0answers
25 views

Doubt in Sutton & Barto's off-policy Monte Carlo control algorithm

The algorithm is described as below: My understanding: In the third last step, we act greedily w.r.t $Q$. Since we use importance sampling, this $Q \approx Q_\pi$. However, in the next step, whenever ...
0
votes
0answers
61 views

How to normalize rewards in REINFORCE?

I'm trying to solve a reinforcement learning problem using a Monte Carlo policy gradient algorithm and, more specifically, REINFORCE, with rewards attributed to individual moves instead of applied to ...
0
votes
0answers
34 views

Defining states and possible actions in Q learning

I am trying to define the number of states and possible actions for a reinforcement learning problem that I want to solve with Q-learning, but I am a bit confused, as I'm totally new to reinforcement ...
1
vote
0answers
18 views

Is there a gentle introduction to reinforcement learning applied to MDPs with continuous state spaces?

I am looking for a gentle introduction (videos, lecture notes, tutorials, books) on reinforcement learning (MDPs) involving continuous states (or very large cardinality of state space). In particular, ...
0
votes
0answers
37 views

Why don't I get the same results of Q-Learning as in Aurélion Géron's Hands-on Machine Learning book?

I noticed something rather intriguing while testing the Deep Q-Network implementation from Aurélion Géron's book Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition; I copy-...
0
votes
0answers
12 views

PPO inputs giving different results in the same environment

I'm using Unity Ml-agents to learn more about reinforcement learning. I've created the most basic environment I could think of, but I'm getting some interesting results. Take a look at the picture ...
0
votes
1answer
28 views

Where can I find the original conference paper that introduced Q-learning and Deep Q-Learning?

I tried searching a lot, but I could neither find the paper that introduced Q-Learning nor the paper that introduced Deep Q Learning. If anyone knows anything about it please do tell me.
1
vote
0answers
14 views

What is the sample complexity of Monte Carlo Exploring Starts in RL?

We can use a model-free Monte Carlo approach to solving an MDP $(S,A,R,P,\gamma)$ with transition dynamics $P$ unknown by estimating Q-values by rolling out trajectories starting from random states $...
0
votes
1answer
32 views

How to interpret the policy gradient expression in reinforcement learning?

I'm currently going through the OpenAI's spinning up introduction course to reinforcement learning. On one of the final sections, they derive an expression for the gradient of the undiscounted return ...
0
votes
0answers
28 views

tanh activation function output is not between -1 and 1 for continuous action PPO

I am using RLlib's (Ray = 1.4.0) PPO policy, and my first layer after the input (Conv layer) is producing a strange output keeping in mind that the activation for the output is a tanh, which I do ...
0
votes
0answers
41 views

RLLib - What exactly do the avail_action and action_embed_size represent? How do they work with the action_mask to phase out invalid actions?

So, I'm fairly new to reinforcement learning and I needed some help/explanations as to what the action_mask and avail_action fields alongside the action_embed_size actually mean in RLlib (the ...
1
vote
1answer
54 views

Why do I get bad results no matter my neural network function approximator for parametrized Q-learning implementation for Contextual Bandits?

I'd like to ask you why, no matter my neural network function approximator for parametrized Q-learning implementation for a Contextual Bandits environment, I'm getting bad results. I don't know if it'...
1
vote
1answer
31 views

How to define a continuous action distribution with a specific range for Reinforcement Learning?

Specifically for continuous control PPO, let's say my action space range is between $X$ (low) and $Y$ (high) and they are all sampled from a Gaussian Action Distribution with mean $\mu$ and standard ...
3
votes
2answers
198 views

How can we find the value function by solving a system of linear equations?

I am following the book "Reinforcement Learning: An Introduction" by Richard Sutton and Andrew Barto, and they give an example of a problem for which the value function can be computed ...
0
votes
1answer
13 views

TD3 sticking to end values

I am using TD3 to train custom gym environment, but the problem is action values stick to the end. Sticking to the end values makes reward negative, to be positive it must find action values somewhere ...
0
votes
0answers
16 views

Multi-agent policy gradient, 1 total reward instead of reward in each step, 2 changing action space

I am new in reinforcement learning and not sure I have the right understanding of multi-agent policy gradient. 1, in my question, each agent has its own action space. When doing the sampling, for each ...
0
votes
0answers
67 views

RLlib's Multi-agent PPO continuous actions turn into nan

After some amount of training on a custom Multi-agent sparse-reward environment using RLlib's (1.4.0) PPO network, I found that my continuous actions turn into nan (explodes?) which is probably caused ...
1
vote
3answers
121 views

Can people use neural networks without providing the set of training data?

It seems that neural networks (NNs) can be applied to supervised learning, unsupervised learning and reinforcement learning. Some people even train neural networks without the set of training data. If ...

1
2
3 4 5
37