Questions tagged [reinforcement-learning]

For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.

Filter by
Sorted by
Tagged with
1 vote
0 answers
12 views

Why slow-changing policy invalidates Double DQN approach in TD3 paper?

In the paper describing TD3 (https://arxiv.org/abs/1802.09477), the authors say that they could not effectively address the Q-learning overestimation bias by using different networks for maximizing ...
Jerry Ding's user avatar
1 vote
1 answer
41 views

Why are these two implementations of the $\epsilon$-greedy policy different?

According to the book Reinforcement Learning An Introduction, the epsilon greedy policy can generally implemented as: $$ \pi(a|s) = \begin{cases} \frac{\epsilon}{|A|} + 1 - \epsilon & \text{if } ...
kklaw's user avatar
  • 143
2 votes
1 answer
94 views

What are the similarities between Q-learning and Value Iteration?

This is the explanation of value iteration in our notes where you keep applying bellman optimality equation till it stops changing and then acting greedily wrt the value function gives the optimal ...
ace239's user avatar
  • 23
0 votes
0 answers
21 views

Using reinforcement learning to optimize a state-based potential function

I am using reinforcement learning for a project, and the objective function I want to optimize is a potential-based function. In other words, it is preferable for the agent to move to positions with ...
LY Omega's user avatar
2 votes
0 answers
37 views

Why does only Deep Q Learning have an overestimation bias?

There is a lot of discussion about the overestimation bias for Deep Q Learning and similar off-policy action value estimation algorithms like DDPG. This is why methods like Double DQN and TD3 were ...
Jerry Ding's user avatar
1 vote
0 answers
52 views

When can we unnest the minimizations/recursions in an value function(bellman optimality equation)?

When reading the following paper(page 4): An Approximate Dynamic Programming Approach for Dual Stochastic Model Predictive Control I could see that they were able to unnest the minimization's in the ...
richard baws's user avatar
0 votes
0 answers
18 views

Can I use Prioritized Learning when the transition probabilities are given?

Can I use Prioritized Learning when the transition probabilities are given? Also, as I can understand, Prioritized Sweeping is suitable when a state space is large. Can I use it when the state space ...
Annaassymeon2's user avatar
1 vote
1 answer
47 views

OpenAI Gym implementation of the delayed rewards

My question is about whether is it possible to implement delayed reward logic within Gym environment. More specifically, I work on ride-pooling RL algorithm, when the action (choice of the parameters ...
Klavdiia Bochenina's user avatar
1 vote
0 answers
22 views

How to represent cards for uno game

I am currently trying to build a DQN agent that plays the game UNO The observation it gets looks like this: ...
Devin Myers's user avatar
0 votes
0 answers
28 views

How to apply DRL to solve a problem that involves mixed discrete-continuous action spaces where the action's size changes over time?

I have a reinforcement learning problem where a possible action is a probability vector $[p_1\ldots,p_n]$ of size $n\in\{1,\ldots,N\}$, where each element $p_i$ of the vector is between $0$ and $1$ ...
zdm's user avatar
  • 301
2 votes
2 answers
58 views

What happens when the probability of either one of the policies is 0 in Importance Sampling?

I have a general question about the methods that use importance sampling in RL. What happens when the probability of either one of the policies is 0?
A J's user avatar
  • 21
3 votes
2 answers
911 views

What is the relation between Dynamic Programming and Reinforcement Learning?

Please forgive me for the implicity of the question, as I recently started studying Reinforcement Learning. I am supposed to study a system where the transition probabilities are known and I have to ...
Annaassymeon2's user avatar
0 votes
1 answer
24 views

Why is my agent stuck on the same action in my Twin Delayed Deep Deterministic Policy Gradient (TD3) program?

I've been tirelessly converting a reinforcement learning program from Python to JavaScript using TensorFlow.js that is running Twin Delayed Deep Deterministic Policy Gradient (TD3). I'm just trying to ...
CloudZero's user avatar
0 votes
0 answers
10 views

When using Reinforcement Learning with Human Feedback to train a transformer, how do I propagate the feedback through the transformer?

I'm basically trying to replicate the processed used to create Chat GPT: Am I supposed to backpropagate? How can I do that when these aren't really errors, but rather ranking several response? Can I ...
Austin Capobianco's user avatar
1 vote
1 answer
52 views

RL agent for autonomous vehicle is able to follow the road but can't avoid crashing at all (Highway-Env / Racetrack Env.)

I coded some deep RL algorithms (DQN and SAC) with tf2/keras to solve an environment where a vehicle needs to follow the track and avoid crashing into one other vehicle (there is only one other ...
rafiqollective's user avatar
1 vote
1 answer
60 views

Could someone give a very simple example of Q-learning in a very small environment? [closed]

I would really like to see an example of Q-learning that I could read, so that I can learn Q-learning from scratch. I read some articles on the internet, but I found it a little difficult to ...
will The J's user avatar
3 votes
1 answer
94 views

Convergence of epsilon greedy policy (with no epsilon decay) using TD Learning?

If I create a policy using the q-values of an epsilon greedy policy using the Sarsa algorithm (not changing the epsilon with each episode), will it converge to the optimal solution to the MDP? I am ...
Prabhjot Singh Rai's user avatar
-1 votes
1 answer
81 views

in simple words, what is the Q-learning algortimn steps? [closed]

I read this article about Q-learning: https://www.learndatasci.com/tutorials/reinforcement-q-learning-scratch-python-openai-gym/ It teaches how to implement the algorithm using the Gym Python library. ...
will The J's user avatar
1 vote
0 answers
17 views

How can I prove that early termination in an MDP state is valid?

I have an MDP $M$ with transition function $p$, states $S$ and a state $s^0 \in S$. $s^0$ has the property that both the most optimistic trajectory (highest expected reward) and the most pessimistic ...
corazza's user avatar
  • 111
0 votes
0 answers
18 views

Can A2C deal with a reward that is decided later than action selection?

I am trying to use a policy gradient based RL algorithm, A2C. However, my training case is slightly different from what typical training tragets are. In my case, a reward is given not immediately ...
user77436's user avatar
1 vote
0 answers
51 views

RL agent focusses too much on early rewards, even with no discounting

How can I guide my RL agent to solve tasks in the correct order? I'm trying to train an agent using reinforcement learning, similar to MuZero. The goal is to solve 4 tasks, A/B/C/D. Each task involves ...
Christopher's user avatar
1 vote
0 answers
58 views

Is this a bandit problem or a MDP?

I am trying to understand if this problem can be casted both as a bandit problem as well as an MDP. Lets assume that we are trying to optimize sales $y_t$ based on investments $x_{1, t}, x_{2, t}$ ...
hugh's user avatar
  • 53
1 vote
0 answers
34 views

When reading on RLHF, I came across this formula but can't break it down

What is the exact meaning of this expression? I'm unsure on the notation. I believe E[R(s)] is expected value of reward of state s, but I'm unsure what the subscript under the E means.
Ryan Marr's user avatar
0 votes
1 answer
60 views

Trading bot with RL, automated actions, nonconvergence

I am playing around with RL to develop a trading bot (using DQN). (Disclaimer: I know, that short term stock movements are near-random and having a bot that is actually useful not likely to happen. ...
Andy's user avatar
  • 1
0 votes
1 answer
31 views

Can single agent trained by RL handle total different tasks?

In the classic "human level control" paper, it writes: We demonstrate that the deep Q-network agent, receiving only the pixels and the game score as inputs, was able to surpass the ...
zhixin's user avatar
  • 15
0 votes
0 answers
17 views

Offline CartPole on infinitely long line?

I am tentatively exploring some RL research that involves doing offline RL on a version of the Gymnasium CartPole where the cart can move on $\mathbb R$, as opposed to the standard version (see link) ...
Novice's user avatar
  • 111
3 votes
0 answers
26 views

Algorithms for average reward reinforcement learning in continuous/general state-action space

I see that discounted reward reinforcement learning has been extensively studied in the literature. However, the average reward metric receives less attention, and it looks like algorithms for this ...
k2pctdn's user avatar
  • 55
1 vote
1 answer
98 views

Reinforcement Learning vs Supervised Learning [duplicate]

I have never tried reinforcement learning in my life. I'm planning to apply it in robotics. I have some experiences using supervised learning mainly deep learning. So, that's mean I will use neural ...
Muhammad Ikhwan Perwira's user avatar
0 votes
0 answers
14 views

Do Bernoulli bandits need a different treatment if the rewards are sparse?

I have a problem where, effectively, my slot machines have very low payout probability (on the order of 1% for the "best" slot machines) and my goal is to minimize the number of actions to ...
Alexander Soare's user avatar
0 votes
0 answers
14 views

Is it possible tofind a subset of F_p for large p such that no solutions exist

I'm aware that neural networks are probably not designed to do that, however asking hypothetically: I have a question regarding the possibility of identifying a subset of $\mathbb{F}_p$ in which a ...
laura's user avatar
  • 101
1 vote
0 answers
31 views

How to find coefficients for a RL-agent environmnet penalty for a buildings district heating peak-shaving problem?

In short, we are trying to use Reinforcement Learning to try to control the heating of a building (district heating) with the input buildings zone temperature, outdoor temperature. To not use the real ...
Tony Karlsson's user avatar
0 votes
1 answer
70 views

Why does my implementation of TD(0) not work?

I am trying to implement TD(0) among other RL Policy Evaluation techniques. I have also implemented the dynamic programming approach for a given model of the world and FV Monte Carlo and EV Monte ...
mavex857's user avatar
0 votes
0 answers
26 views

Multi-Agent DQN not learning for Clean Up Game - Reward slowly decreasing

The environment of the Clean Up game is simple: in a 25*18 grid world, there's dirt spawning on the left side and apples spawning on the other. Agents get a +1 reward for eating an apple (by stepping ...
Charles's user avatar
1 vote
1 answer
46 views

How to plot average return vs step figures in reinforcement learning?

I know return is total discounted reward of an episode,so it is easy to plot return vs. episode.But in many papers,they provide figures of average return vs step,like this: Based on my knowledge,...
waylone's user avatar
  • 11
0 votes
0 answers
93 views

Clarification on Formulation of a DP Problem

I am self-teaching reinforcement learning and for now I am trying to solve the following DP problem: A driver is looking for inexpensive parking on the way to his destination. The parking area ...
ArGenya's user avatar
0 votes
0 answers
28 views

Deep Reinforcement Learning that takes action from two different sets

I am working on a problem where I want to schedule multiple activities (a1, a2, a3, ... aN) requiring different resource types ...
zeeshan's user avatar
2 votes
1 answer
43 views

UCB, Thompson sampling etc seems myopic/greedy for bandits?

When considering multi-armed bandits in different formats, UCB, $\epsilon$-greedy, thompson sampling etc seems so greedy/myopic in the sense that it solely considers reward for the current timestep. ...
hugh's user avatar
  • 53
1 vote
0 answers
26 views

Intuition behind why Posterior Sampling Lemma holds?

Posterior Sampling Lemma was introduced in the (More) Efficient RL via Posterior Sampling and looks like this. $M^*$ here is the true MDP while $M_k$ is the MDP sampled from the posterior in episode $...
pecey's user avatar
  • 333
0 votes
1 answer
51 views

Discount factors in REINFORCE Algorithm: Difference in two definitions

I am a bit confused between two definitions of the Vanilla REINFORCE algorithm. The first one is in the following (from this page: https://stjohngrimbly.com/model-free-RL/): Here, at every step in an ...
Ufuk Can Bicici's user avatar
0 votes
0 answers
13 views

What reinforcement learning method tries to learn the model? Is it better?

So I haven't read through every reinforcement learning algorithm, so I am curious. The model-free methods I have read about so far basically just try things and see what works and what doesn't. They ...
Anish Kommireddy's user avatar
3 votes
0 answers
51 views

Why policy gradient theorem has two different forms?

I have been studying policy gradients recently but found different expositions from different sources, which greatly confused me. From the book "Reinforcement Learning: an Introduction (Sutton &...
Yuxiang Wei's user avatar
0 votes
1 answer
91 views

My first experience with gym environment has raised many questions, and I need some guidance [closed]

As I'm new to the AI/ML field, I'm still learning from various online materials. In this particular instance, I've been studying the Reinforcement Learning tutorial by deeplizard, specifically ...
Boris L.'s user avatar
  • 109
0 votes
0 answers
24 views

Policy Gradient in Partial Observability

Let $\pi_{\theta}$ be a policy. Then, I was able to follow through the proof of: $\nabla_\theta J=\mathbb{E}_{\tau\sim\pi_{\theta}}[\Sigma_{i=1}^T \nabla_\theta log(p_{\theta}(a_i|s_i)R(\tau)]$, where ...
A J's user avatar
  • 21
2 votes
2 answers
68 views

How does the agent derive the state from the observations in Reinforcement Learning?

I am not at all experienced in RL programs. I have been reading up about them recently, and I learned about states. I thought of it as all the inputs to the RL. Then I stumbled across observations ...
Rocket Man's user avatar
0 votes
1 answer
60 views

Understanding KL Stopping and KL Cutoff for the PPO algorithm

I am reading a couple of review papers to optimize the PPO algorithm. It seems like the review papers are saying the same thing but used slightly different terms. Could someone please tell if the ...
desert_ranger's user avatar
4 votes
1 answer
83 views

Modern reinforcement learning for video game NPCs

Recently, I have been reading about the 1996 artificial life game 'Creatures'. The game features NPCs called 'Norns' that use reinforcement learning to learn continuously through interactions with the ...
akliyen's user avatar
  • 43
0 votes
0 answers
19 views

Reinforcement learning: enviroment changes a lot while DQN is still computing actions

I'm implementing a agent to dodge skillshots. Here's the problem: while DQN is computing action based on observation, enviroment changed meanwhile. Since those skillshots are quite fast, the ...
口乞丿丶's user avatar
0 votes
0 answers
22 views

Query modification for search using AI

I have a problem statement that I'm struggling to formulate as a machine learning framework. There is a huge client database of documents - we're trying to come up with an efficient way of querying ...
user9343456's user avatar
1 vote
0 answers
18 views

How does recurrent neural network implement model based RL system purely in its activation dynamics (in blackbox meta-rl setting)?

I have read these papers "learning to reinforcement learn" and "PFC as meta RL system". The authors claim that when RNN is trained on multiple tasks from a task distribution using ...
veerendra's user avatar
1 vote
1 answer
99 views

What kind of observation state would you give for that environment?

I'm making a new environment where I have two sphere (one above the other) in a 2D plan. I would like some advice on what observation state I should give to my RL. Today I have given the following: ...
CyDevos's user avatar
  • 145

1
2 3 4 5
48