Questions tagged [reinforcement-learning]
For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.
2,378
questions
1
vote
0
answers
12
views
Why slow-changing policy invalidates Double DQN approach in TD3 paper?
In the paper describing TD3 (https://arxiv.org/abs/1802.09477), the authors say that they could not effectively address the Q-learning overestimation bias by using different networks for maximizing ...
- 131
1
vote
1
answer
41
views
Why are these two implementations of the $\epsilon$-greedy policy different?
According to the book Reinforcement Learning An Introduction, the epsilon greedy policy can generally implemented as:
$$
\pi(a|s) =
\begin{cases}
\frac{\epsilon}{|A|} + 1 - \epsilon & \text{if } ...
- 143
2
votes
1
answer
94
views
What are the similarities between Q-learning and Value Iteration?
This is the explanation of value iteration in our notes where you keep applying bellman optimality equation till it stops changing and then acting greedily wrt the value function gives the optimal ...
- 23
0
votes
0
answers
21
views
Using reinforcement learning to optimize a state-based potential function
I am using reinforcement learning for a project, and the objective function I want to optimize is a potential-based function. In other words, it is preferable for the agent to move to positions with ...
- 1
2
votes
0
answers
37
views
Why does only Deep Q Learning have an overestimation bias?
There is a lot of discussion about the overestimation bias for Deep Q Learning and similar off-policy action value estimation algorithms like DDPG. This is why methods like Double DQN and TD3 were ...
- 131
1
vote
0
answers
52
views
When can we unnest the minimizations/recursions in an value function(bellman optimality equation)?
When reading the following paper(page 4): An Approximate Dynamic Programming Approach
for Dual Stochastic Model Predictive Control
I could see that they were able to unnest the minimization's in the ...
- 11
0
votes
0
answers
18
views
Can I use Prioritized Learning when the transition probabilities are given?
Can I use Prioritized Learning when the transition probabilities are given?
Also, as I can understand, Prioritized Sweeping is suitable when a state space is large. Can I use it when the state space ...
1
vote
1
answer
47
views
OpenAI Gym implementation of the delayed rewards
My question is about whether is it possible to implement delayed reward logic within Gym environment.
More specifically, I work on ride-pooling RL algorithm, when the action (choice of the parameters ...
1
vote
0
answers
22
views
How to represent cards for uno game
I am currently trying to build a DQN agent that plays the game UNO
The observation it gets looks like this:
...
- 11
0
votes
0
answers
28
views
How to apply DRL to solve a problem that involves mixed discrete-continuous action spaces where the action's size changes over time?
I have a reinforcement learning problem where a possible action is a probability vector $[p_1\ldots,p_n]$ of size $n\in\{1,\ldots,N\}$, where each element $p_i$ of the vector is between $0$ and $1$ ...
- 301
2
votes
2
answers
58
views
What happens when the probability of either one of the policies is 0 in Importance Sampling?
I have a general question about the methods that use importance sampling in RL. What happens when the probability of either one of the policies is 0?
- 21
3
votes
2
answers
911
views
What is the relation between Dynamic Programming and Reinforcement Learning?
Please forgive me for the implicity of the question, as I recently started studying Reinforcement Learning.
I am supposed to study a system where the transition probabilities are known and I have to ...
0
votes
1
answer
24
views
Why is my agent stuck on the same action in my Twin Delayed Deep Deterministic Policy Gradient (TD3) program?
I've been tirelessly converting a reinforcement learning program from Python to JavaScript using TensorFlow.js that is running Twin Delayed Deep Deterministic Policy Gradient (TD3). I'm just trying to ...
0
votes
0
answers
10
views
When using Reinforcement Learning with Human Feedback to train a transformer, how do I propagate the feedback through the transformer?
I'm basically trying to replicate the processed used to create Chat GPT:
Am I supposed to backpropagate? How can I do that when these aren't really errors, but rather ranking several response? Can I ...
1
vote
1
answer
52
views
RL agent for autonomous vehicle is able to follow the road but can't avoid crashing at all (Highway-Env / Racetrack Env.)
I coded some deep RL algorithms (DQN and SAC) with tf2/keras to solve an environment where a vehicle needs to follow the track and avoid crashing into one other vehicle (there is only one other ...
1
vote
1
answer
60
views
Could someone give a very simple example of Q-learning in a very small environment? [closed]
I would really like to see an example of Q-learning that I could read, so that I can learn Q-learning from scratch. I read some articles on the internet, but I found it a little difficult to ...
- 233
3
votes
1
answer
94
views
Convergence of epsilon greedy policy (with no epsilon decay) using TD Learning?
If I create a policy using the q-values of an epsilon greedy policy using the Sarsa algorithm (not changing the epsilon with each episode), will it converge to the optimal solution to the MDP? I am ...
-1
votes
1
answer
81
views
in simple words, what is the Q-learning algortimn steps? [closed]
I read this article about Q-learning:
https://www.learndatasci.com/tutorials/reinforcement-q-learning-scratch-python-openai-gym/
It teaches how to implement the algorithm using the Gym Python library.
...
- 233
1
vote
0
answers
17
views
How can I prove that early termination in an MDP state is valid?
I have an MDP $M$ with transition function $p$, states $S$ and a state $s^0 \in S$. $s^0$ has the property that both the most optimistic trajectory (highest expected reward) and the most pessimistic ...
- 111
0
votes
0
answers
18
views
Can A2C deal with a reward that is decided later than action selection?
I am trying to use a policy gradient based RL algorithm, A2C. However, my training case is slightly different from what typical training tragets are. In my case, a reward is given not immediately ...
1
vote
0
answers
51
views
RL agent focusses too much on early rewards, even with no discounting
How can I guide my RL agent to solve tasks in the correct order?
I'm trying to train an agent using reinforcement learning, similar to MuZero. The goal is to solve 4 tasks, A/B/C/D. Each task involves ...
- 133
1
vote
0
answers
58
views
Is this a bandit problem or a MDP?
I am trying to understand if this problem can be casted both as a bandit problem as well as an MDP.
Lets assume that we are trying to optimize sales $y_t$ based on investments $x_{1, t}, x_{2, t}$ ...
- 53
1
vote
0
answers
34
views
When reading on RLHF, I came across this formula but can't break it down
What is the exact meaning of this expression? I'm unsure on the notation.
I believe E[R(s)] is expected value of reward of state s, but I'm unsure what the subscript under the E means.
- 11
0
votes
1
answer
60
views
Trading bot with RL, automated actions, nonconvergence
I am playing around with RL to develop a trading bot (using DQN). (Disclaimer: I know, that short term stock movements are near-random and having a bot that is actually useful not likely to happen. ...
- 1
0
votes
1
answer
31
views
Can single agent trained by RL handle total different tasks?
In the classic "human level control" paper, it writes:
We demonstrate that the deep Q-network agent, receiving only the pixels and the game score as inputs, was able to surpass the ...
- 15
0
votes
0
answers
17
views
Offline CartPole on infinitely long line?
I am tentatively exploring some RL research that involves doing offline RL on a version of the Gymnasium CartPole where the cart can move on $\mathbb R$, as opposed to the standard version (see link) ...
- 111
3
votes
0
answers
26
views
Algorithms for average reward reinforcement learning in continuous/general state-action space
I see that discounted reward reinforcement learning has been extensively studied in the literature. However, the average reward metric receives less attention, and it looks like algorithms for this ...
- 55
1
vote
1
answer
98
views
Reinforcement Learning vs Supervised Learning [duplicate]
I have never tried reinforcement learning in my life.
I'm planning to apply it in robotics.
I have some experiences using supervised learning mainly deep learning. So, that's mean I will use neural ...
0
votes
0
answers
14
views
Do Bernoulli bandits need a different treatment if the rewards are sparse?
I have a problem where, effectively, my slot machines have very low payout probability (on the order of 1% for the "best" slot machines) and my goal is to minimize the number of actions to ...
- 1,319
0
votes
0
answers
14
views
Is it possible tofind a subset of F_p for large p such that no solutions exist
I'm aware that neural networks are probably not designed to do that, however asking hypothetically:
I have a question regarding the possibility of identifying a subset of $\mathbb{F}_p$ in which a ...
- 101
1
vote
0
answers
31
views
How to find coefficients for a RL-agent environmnet penalty for a buildings district heating peak-shaving problem?
In short, we are trying to use Reinforcement Learning to try to control the heating of a building (district heating) with the input buildings zone temperature, outdoor temperature. To not use the real ...
0
votes
1
answer
70
views
Why does my implementation of TD(0) not work?
I am trying to implement TD(0) among other RL Policy Evaluation techniques.
I have also implemented the dynamic programming approach for a given model of the world and FV Monte Carlo and EV Monte ...
- 21
0
votes
0
answers
26
views
Multi-Agent DQN not learning for Clean Up Game - Reward slowly decreasing
The environment of the Clean Up game is simple: in a 25*18 grid world, there's dirt spawning on the left side and apples spawning on the other. Agents get a +1 reward for eating an apple (by stepping ...
- 1
1
vote
1
answer
46
views
How to plot average return vs step figures in reinforcement learning？
I know return is total discounted reward of an episode,so it is easy to plot return vs. episode.But in many papers,they provide figures of average return vs step,like this:
Based on my knowledge,...
- 11
0
votes
0
answers
93
views
Clarification on Formulation of a DP Problem
I am self-teaching reinforcement learning and for now I am trying to solve the following DP problem:
A driver is looking for inexpensive parking on the way to his destination. The parking
area ...
- 1
0
votes
0
answers
28
views
Deep Reinforcement Learning that takes action from two different sets
I am working on a problem where I want to schedule multiple activities (a1, a2, a3, ... aN) requiring different resource types ...
- 1
2
votes
1
answer
43
views
UCB, Thompson sampling etc seems myopic/greedy for bandits?
When considering multi-armed bandits in different formats, UCB, $\epsilon$-greedy, thompson sampling etc seems so greedy/myopic in the sense that it solely considers reward for the current timestep.
...
- 53
1
vote
0
answers
26
views
Intuition behind why Posterior Sampling Lemma holds?
Posterior Sampling Lemma was introduced in the (More) Efficient RL via Posterior Sampling and looks like this.
$M^*$ here is the true MDP while $M_k$ is the MDP sampled from the posterior in episode $...
- 333
0
votes
1
answer
51
views
Discount factors in REINFORCE Algorithm: Difference in two definitions
I am a bit confused between two definitions of the Vanilla REINFORCE algorithm.
The first one is in the following (from this page: https://stjohngrimbly.com/model-free-RL/):
Here, at every step in an ...
- 101
0
votes
0
answers
13
views
What reinforcement learning method tries to learn the model? Is it better?
So I haven't read through every reinforcement learning algorithm, so I am curious. The model-free methods I have read about so far basically just try things and see what works and what doesn't. They ...
- 111
3
votes
0
answers
51
views
Why policy gradient theorem has two different forms?
I have been studying policy gradients recently but found different expositions from different sources, which greatly confused me. From the book "Reinforcement Learning: an Introduction (Sutton &...
- 31
0
votes
1
answer
91
views
My first experience with gym environment has raised many questions, and I need some guidance [closed]
As I'm new to the AI/ML field, I'm still learning from various online materials. In this particular instance, I've been studying the Reinforcement Learning tutorial by deeplizard, specifically ...
- 109
0
votes
0
answers
24
views
Policy Gradient in Partial Observability
Let $\pi_{\theta}$ be a policy. Then, I was able to follow through the proof of: $\nabla_\theta J=\mathbb{E}_{\tau\sim\pi_{\theta}}[\Sigma_{i=1}^T \nabla_\theta log(p_{\theta}(a_i|s_i)R(\tau)]$, where ...
- 21
2
votes
2
answers
68
views
How does the agent derive the state from the observations in Reinforcement Learning?
I am not at all experienced in RL programs. I have been reading up about them recently, and I learned about states. I thought of it as all the inputs to the RL. Then I stumbled across observations ...
- 21
0
votes
1
answer
60
views
Understanding KL Stopping and KL Cutoff for the PPO algorithm
I am reading a couple of review papers to optimize the PPO algorithm. It seems like the review papers are saying the same thing but used slightly different terms. Could someone please tell if the ...
- 608
4
votes
1
answer
83
views
Modern reinforcement learning for video game NPCs
Recently, I have been reading about the 1996 artificial life game 'Creatures'. The game features NPCs called 'Norns' that use reinforcement learning to learn continuously through interactions with the ...
- 43
0
votes
0
answers
19
views
Reinforcement learning: enviroment changes a lot while DQN is still computing actions
I'm implementing a agent to dodge skillshots. Here's the problem: while DQN is computing action based on observation, enviroment changed meanwhile. Since those skillshots are quite fast, the ...
- 1
0
votes
0
answers
22
views
Query modification for search using AI
I have a problem statement that I'm struggling to formulate as a machine learning framework. There is a huge client database of documents - we're trying to come up with an efficient way of querying ...
- 151
1
vote
0
answers
18
views
How does recurrent neural network implement model based RL system purely in its activation dynamics (in blackbox meta-rl setting)?
I have read these papers "learning to reinforcement learn" and "PFC as meta RL system". The authors claim that when RNN is trained on multiple tasks from a task distribution using ...
- 19
1
vote
1
answer
99
views
What kind of observation state would you give for that environment?
I'm making a new environment where I have two sphere (one above the other) in a 2D plan.
I would like some advice on what observation state I should give to my RL.
Today I have given the following:
...
- 145