Questions tagged [reinforcement-learning]
For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.
2,280
questions
0
votes
0
answers
40
views
Using reinforcement learning with an undefined set of actions and observations
I am new to RL and want to build a tool that will make certain decisions in order to navigate through a web page to which I own the code. My goal is the agent to take different actions based on the ...
- 101
0
votes
1
answer
23
views
Q learning: Action is combination of two actions
Suppose in my example I want an agent to learn a behavior that is made up of a combination of actions.
So consider the following example with a tamagotchi like game: There are 5 pets and 3 actions ...
- 101
0
votes
0
answers
21
views
Role of $f$ Target Network in DDPG
I am trying to create a variant of DDPG in MATLAB that has no action-value $\langle Q \rangle$ net, but that instead works with networks $\langle V \rangle, \langle f \rangle, \langle r \rangle$, and ...
2
votes
0
answers
25
views
How can i tinker my neural network to learn stronger on rare events?
I am training a neural network on a regression problem.
Most of the time the actual y (label) has the same value (say ~0.2) and only in rare cases the actual y is very different (say 2.0 or -2.0)
...
- 21
0
votes
0
answers
13
views
Bellmann Error with Time Step
I define the return $G$ as the discrete time integral of the reward $G = \Delta t \sum_{t = 0, ..., T}r_t$. In supposing $\Delta t = 1$ throughout, my instructor wrote the following formula for the ...
0
votes
0
answers
25
views
How to use "states" as vectors in Q-Learning?
How do I use states as vectors in Q-Learning or any other RL Algorithms. Let's say I have state as a vector with probabilities[0,1], and I have to take an action if the state is valid (with ...
1
vote
3
answers
58
views
Where would I start if I wanted to create an AI agent to play a 2d game?
I am keen on creating a little project that can play a fairly basic 2D game (more complex than say, snake but not as complex as mario kart) and would like some pointers on where to begin. I'm entirely ...
- 11
-1
votes
0
answers
17
views
Why is even q estimation reduced but the agent cannot predict correctly after training? (offline q learning)
I am going to implement Algorithm 1 in this paper.
When I train the agent, it was gradually reduced the q estimate and, after training agent cannot predict correctly. What is the reason for that?
- 1
0
votes
0
answers
23
views
Problem of extremely varied reward value in DDQN
I am trying to train my model by DDQN agent after creating a customized environment in gym. I am stating my hyper-parameters and other details here.
...
- 101
-2
votes
0
answers
18
views
Soft actor critic using the tabular setting and large state space
How we can solve an ordering system with the soft actor critic algorithm?
- 1
2
votes
1
answer
119
views
Why is soft actor critic an off policy scheme?
I am struggling to understand what makes a scheme on-policy or off-policy. From what I have read, we can say that deep Q-learning is off-policy because we use a different policy like $\epsilon$-greedy ...
- 35
0
votes
0
answers
14
views
An example of a non-sequentially consistent heuristic?
Definition: For the dynamical system, $x_{k+1}=f(x_k,u_k)$, we say that a heuristic is sequentially consistent if it has the property that when it generates the sequence
$\{x_k, u_k, x_{k+1}, u_{k+1}, ...
- 281
0
votes
0
answers
13
views
Is there any dedicated library/software for creating backup diagrams (as used in Sutton and Barto's Reinforcement Learning book)?
I am currently working through Sutton and Barto's Reinforcement Learning book.
I have found that backup diagrams help me a lot in wrapping my head around the presented algorithms and concepts.
How ...
- 1
0
votes
0
answers
12
views
Best fitness function for car driving around a procedural race track?
I am using the NEAT algorithm to power a car around a procedural race track. Each generation, a new track is created to avoid the car overfitting the environment.
The aim is for the car to drive ...
1
vote
1
answer
46
views
It is not clear why sequential improvement is violated in the constructed rollout algorithm
Below is an example (p.89) from "RL and Optimal Control" book by D.Bertsekas on the construction of a a case study where the rollout algorithm is worse than the base heuristic on which the ...
- 281
2
votes
1
answer
36
views
What is the difference between "base heuristic" and a "rollout algorithm" based on this base heuristic?
In the following pages (p.84-85) of RL and Optimal Control book by D.Bertsekas,
he is talking about base heuristic" and a "rollout algorithm" based on this base heuristic.
However, I ...
- 281
0
votes
0
answers
17
views
Penalizing invalid action and ending episode early causes agent to end episode with invalid action in a non-goal state?
Domain: start at initial position, navigate to goal-position with N,S,E,W actions.
Algorithm: PPO
Libs: custom gym-env, stable baselines3
Penalties:
valid step: -1 (promote shortest path)
INvalid step:...
0
votes
0
answers
27
views
Reinforcement Learning (gymnasium's FrozenLake-v1) using Spiking Neural Networks (BindsNet)
I'm new to reinforcement learning. I'm trying to solve the FrozenLake-v1 game using OpenAI's gymnasium learning environment and BindsNet, which is a library to simulate Spiking Neural Networks using ...
- 1
0
votes
0
answers
20
views
What reinforcement learning algorithm should I use for the following problem?
Environment
I have a static timeseries environment meaning the environment is the same.
This problem is a multi armed bandit problem.
Time
t0
t1
t2
State
s0
s1
s2
Score
10
0.1
0.2
Class
1
0
0
...
- 1
0
votes
1
answer
33
views
Are platformer games, with the camera centred on the character, examples of egocentric vision?
An example may be CoinRun, where the character sprite is always centred in the camera view, while the environment moves as a result of player input.
To me, this sounds like egocentric vision, although ...
- 271
0
votes
0
answers
11
views
Request for assistance with converting legal contracts to environment for DQN
I want to convert the Extractive QA task as a Reinforcement Learning Problem Statement. So I want to integrate NLP problem into Reinforcement Learning and see if my results were achieving better when ...
- 111
0
votes
0
answers
22
views
DQN does not learn the CartPole problem
I've reviewed my code several times ... I can't spot the problem. When I train the agent, it doesn't learn anything. I guess the problem is the way I compute the loss function but I don't know any ...
1
vote
1
answer
75
views
Why is Soft Q Learning not an Actor Critic method?
I've been reading these two papers from Haarnoja et. al.:
Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement
Learning with a Stochastic Actor
Reinforcement Learning with Deep Energy-...
0
votes
0
answers
41
views
Looking for a reinforcement learning algorithm that deals well with a model-based, curiosity-driven approach for chess AI
I am a software engineer that meddled with machine learning (classifiers) during my thesis.
After being out of it for a while I decided I want to try and do a neural network project to learn from, ...
- 119
0
votes
0
answers
13
views
Is there any benefit to using different polyak numbers for target policy and target Q-function parameters?
In the context of RL and DDPG, it seems polyak averaging is used with reference to two things; the target policy, and the target Q parameters. I know that these hyper parameters can be a bit finicky ...
- 133
0
votes
0
answers
26
views
A2C unable to solve Cartpole
I have coded my own A2C implementation using PyTorch. However, despite having followed the algorithm pseudo-code from several sources, my implementation is not able to achieve a proper Cartpole ...
- 16
0
votes
1
answer
39
views
Time index in TD(0) return
in TD(0). for the return we have:
$G_{t:t+1}=r_{t+1}+\gamma v_t(s_{t+1})$.
Why is the time index on right hand side in $v$ is $t$?
- 281
1
vote
0
answers
40
views
Prioritized experience replay correction with off-policy estimators
Prioritized exeperience replay (PER) biases the sampling and introduces importance sampling (IS) correction to the Q-function update.
Weights are $w = \frac{1}{N P}^\beta$, where $N$ is the batch size ...
- 111
1
vote
1
answer
28
views
Confusion in subscript for n-step TD(0)
This is n-step TD(0) update rule:
$v_{k+n}(x_k)=v_{k+n-1}(x_k)+\alpha [g_{k:k+n}-v_{k+n-1}(x_k)]$
Why is the subscript on the left hand-side of equation "k+n", not "k+n-1"? Does ...
- 281
0
votes
0
answers
25
views
Understanding SARSA with binary features and function approximation
I'm reading the Sutton and Barto Book on Reinforcement Learning (for oral exam preparation).
However, I'm stuck on the algorithm for SARSA($\lambda$) with binary features and linear function ...
- 1
0
votes
0
answers
25
views
Very high dimensional optimization with large budget, requiring high quality solutions
What would be theoretically the best performing optimization algorithm(s) in this case?
Very high dimensional problem: 250-500 parameters
Goal is to obtain very high quality solutions, not just "...
0
votes
0
answers
15
views
Dealing with long running tasks in Q Learning
Assuming Q Learning is applied not to low level behaviors (such as taking a step; drawing/playing a card; moving one piece on a board), but rather longer running high-level behaviors (e.g. Moving from ...
- 1
0
votes
0
answers
22
views
1
vote
0
answers
9
views
Representing non-fixed state-spaces in RL algorithms
How I can define the state of an environment where I don’t have a fixed representation for states? E.g. the state at one step is a list of size 5 and in another state is a list of size 6. What is the ...
0
votes
0
answers
79
views
Why do we need RL in RLHF?
In RLHF, the reward function is a neural network. This means we can compute its gradients cheaply and accurately through backpropagation. Now, we want to find a policy that maximizes reward (see https:...
- 156
0
votes
1
answer
51
views
What should I do, reinforcement learning agent gives different result on every train?
I'm using PPO+LSTM to create a trading bot. The agent is trained on 3 years of data and tested on 1 year. Every time I train the agent with same set of hyper-parameters, I get very different results ...
- 1
1
vote
1
answer
35
views
How to structure the input data for non-vision deep reinforcement learning?
I am currently designing a custom gym environment that is based on sensor data and I struggle a bit with structuring the data and designing the model. Virtually every resource I find online is kind of ...
- 13
1
vote
1
answer
49
views
How to account for a numeric variable in a state in RL?
I am new to reinforcement learning and having a hard time making the leap from tutorials to real world problems.
I'm trying to figure out how to deal with a board game with 9 squares each square ...
- 13
1
vote
1
answer
86
views
Where is the problem: in batch TD(0) algorithm or in the code to solve AB problem in Sutton-Barto RL book?
Here is the batch TD(0) algorithm:
Here is the AB example I want to solve using batch TD(0):
And finally here is my Matlab code:
% eps1: A 0 B 0
% eps2: B 1
% eps3: B 1
% eps4: B 1
% eps5: B 1
% ...
- 281
0
votes
2
answers
61
views
MCTS players keep replaying identical games
I am currently training a self-playing Monte-Carlo-Tree-Search (MCTS) algorithm with a neural network prior, and it seems to be working pretty well.
However one problem I have is when I compare my new ...
- 101
0
votes
1
answer
47
views
Why does $\alpha=1$ mean batch MC Learning?
Here is a part of slide 4 from the link:
https://tao.lri.fr/tiki-download_wiki_attachment.php?attId=1683
Why does $\alpha=1$ mean batch MC Learning? I do not see this clearly when I compare with ...
- 281
2
votes
0
answers
50
views
Can Q-learning and other RL algorithms solve CNF SAT?
I encountered a question about solving CNF SAT using reinforcement learning: A state is a partial substitution to the variables, and each action is choosing an empty variable and set its value (to <...
- 121
0
votes
0
answers
21
views
How (if possible at all) the rewards (from the reinforcement learning) can be used to generate the data for the supervised learning?
How (if possible at all) rewards (from reinforcement learning) can be used to generate data for supervised learning? This is very topical question, because human feedback usually comes in the form or ...
- 803
0
votes
0
answers
25
views
Non-deterministic play in reinforcement learning
I'm training a reinforcement learning agent on simple two-player games (I'm doing Q-learning). I noticed that for a given state the value of each action is usually different. Thus, the greedy policy ...
- 233
0
votes
0
answers
29
views
How to increase exploration in IMPALA?
I am using the RLLib library for applying IMPALA algorithm to my environment. I want to know which variables in https://docs.ray.io/en/latest/_modules/ray/rllib/algorithms/impala/impala.html#...
- 113
2
votes
1
answer
79
views
Is it always a good idea to use deterministic policies during testing?
I frequently see people setting deterministic = True while testing an RL algorithm. But is this the right approach? For instance, what happens if the agent plays rock, paper, and scissors? In this ...
- 478
2
votes
1
answer
27
views
Which Q function do we use to make our policy when using double Q learning?
I know this might be arbitrary, but I couldn't find any good information on this. As we update 2 q function in double q learning I was curios whether we average, or sum them together to get our policy....
- 23
0
votes
0
answers
24
views
How to train a neural network in a RL environment to learn the optimal strategy in tic tac toe?
I have just been diving into machine learning since last week. I'm trying to create a neural network that learns to play tic tac toe optimally in an RL environment. So far with very little success. ...
- 101
3
votes
1
answer
111
views
What is the motivation for using Q-Learning in RL?
In Spinning Up by OpenAI, it says the following regarding policy optimization methods and Q-Learning as ways of getting a good policy for RL.
Trade-offs Between Policy Optimization and Q-Learning. ...
- 133
1
vote
1
answer
136
views
Proof of convergence of TD(0) algorithm
I am looking for a proof of the following tabular TD(0) algorithm:
However, I can only find proofs with the more general TD($\lambda$) algorithm and I am having problems understanding them. In ...
- 11