Questions tagged [reinforcement-learning]

For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.

Filter by
Sorted by
Tagged with
0 votes
0 answers
40 views

Using reinforcement learning with an undefined set of actions and observations

I am new to RL and want to build a tool that will make certain decisions in order to navigate through a web page to which I own the code. My goal is the agent to take different actions based on the ...
  • 101
0 votes
1 answer
23 views

Q learning: Action is combination of two actions

Suppose in my example I want an agent to learn a behavior that is made up of a combination of actions. So consider the following example with a tamagotchi like game: There are 5 pets and 3 actions ...
  • 101
0 votes
0 answers
21 views

Role of $f$ Target Network in DDPG

I am trying to create a variant of DDPG in MATLAB that has no action-value $\langle Q \rangle$ net, but that instead works with networks $\langle V \rangle, \langle f \rangle, \langle r \rangle$, and ...
2 votes
0 answers
25 views

How can i tinker my neural network to learn stronger on rare events?

I am training a neural network on a regression problem. Most of the time the actual y (label) has the same value (say ~0.2) and only in rare cases the actual y is very different (say 2.0 or -2.0) ...
0 votes
0 answers
13 views

Bellmann Error with Time Step

I define the return $G$ as the discrete time integral of the reward $G = \Delta t \sum_{t = 0, ..., T}r_t$. In supposing $\Delta t = 1$ throughout, my instructor wrote the following formula for the ...
0 votes
0 answers
25 views

How to use "states" as vectors in Q-Learning?

How do I use states as vectors in Q-Learning or any other RL Algorithms. Let's say I have state as a vector with probabilities[0,1], and I have to take an action if the state is valid (with ...
1 vote
3 answers
58 views

Where would I start if I wanted to create an AI agent to play a 2d game?

I am keen on creating a little project that can play a fairly basic 2D game (more complex than say, snake but not as complex as mario kart) and would like some pointers on where to begin. I'm entirely ...
-1 votes
0 answers
17 views

Why is even q estimation reduced but the agent cannot predict correctly after training? (offline q learning)

I am going to implement Algorithm 1 in this paper. When I train the agent, it was gradually reduced the q estimate and, after training agent cannot predict correctly. What is the reason for that?
  • 1
0 votes
0 answers
23 views

Problem of extremely varied reward value in DDQN

I am trying to train my model by DDQN agent after creating a customized environment in gym. I am stating my hyper-parameters and other details here. ...
-2 votes
0 answers
18 views

Soft actor critic using the tabular setting and large state space

How we can solve an ordering system with the soft actor critic algorithm?
2 votes
1 answer
119 views

Why is soft actor critic an off policy scheme?

I am struggling to understand what makes a scheme on-policy or off-policy. From what I have read, we can say that deep Q-learning is off-policy because we use a different policy like $\epsilon$-greedy ...
0 votes
0 answers
14 views

An example of a non-sequentially consistent heuristic?

Definition: For the dynamical system, $x_{k+1}=f(x_k,u_k)$, we say that a heuristic is sequentially consistent if it has the property that when it generates the sequence $\{x_k, u_k, x_{k+1}, u_{k+1}, ...
0 votes
0 answers
13 views

Is there any dedicated library/software for creating backup diagrams (as used in Sutton and Barto's Reinforcement Learning book)?

I am currently working through Sutton and Barto's Reinforcement Learning book. I have found that backup diagrams help me a lot in wrapping my head around the presented algorithms and concepts. How ...
  • 1
0 votes
0 answers
12 views

Best fitness function for car driving around a procedural race track?

I am using the NEAT algorithm to power a car around a procedural race track. Each generation, a new track is created to avoid the car overfitting the environment. The aim is for the car to drive ...
1 vote
1 answer
46 views

It is not clear why sequential improvement is violated in the constructed rollout algorithm

Below is an example (p.89) from "RL and Optimal Control" book by D.Bertsekas on the construction of a a case study where the rollout algorithm is worse than the base heuristic on which the ...
2 votes
1 answer
36 views

What is the difference between "base heuristic" and a "rollout algorithm" based on this base heuristic?

In the following pages (p.84-85) of RL and Optimal Control book by D.Bertsekas, he is talking about base heuristic" and a "rollout algorithm" based on this base heuristic. However, I ...
0 votes
0 answers
17 views

Penalizing invalid action and ending episode early causes agent to end episode with invalid action in a non-goal state?

Domain: start at initial position, navigate to goal-position with N,S,E,W actions. Algorithm: PPO Libs: custom gym-env, stable baselines3 Penalties: valid step: -1 (promote shortest path) INvalid step:...
0 votes
0 answers
27 views

Reinforcement Learning (gymnasium's FrozenLake-v1) using Spiking Neural Networks (BindsNet)

I'm new to reinforcement learning. I'm trying to solve the FrozenLake-v1 game using OpenAI's gymnasium learning environment and BindsNet, which is a library to simulate Spiking Neural Networks using ...
0 votes
0 answers
20 views

What reinforcement learning algorithm should I use for the following problem?

Environment I have a static timeseries environment meaning the environment is the same. This problem is a multi armed bandit problem. Time t0 t1 t2 State s0 s1 s2 Score 10 0.1 0.2 Class 1 0 0 ...
0 votes
1 answer
33 views

Are platformer games, with the camera centred on the character, examples of egocentric vision?

An example may be CoinRun, where the character sprite is always centred in the camera view, while the environment moves as a result of player input. To me, this sounds like egocentric vision, although ...
0 votes
0 answers
11 views

Request for assistance with converting legal contracts to environment for DQN

I want to convert the Extractive QA task as a Reinforcement Learning Problem Statement. So I want to integrate NLP problem into Reinforcement Learning and see if my results were achieving better when ...
0 votes
0 answers
22 views

DQN does not learn the CartPole problem

I've reviewed my code several times ... I can't spot the problem. When I train the agent, it doesn't learn anything. I guess the problem is the way I compute the loss function but I don't know any ...
1 vote
1 answer
75 views

Why is Soft Q Learning not an Actor Critic method?

I've been reading these two papers from Haarnoja et. al.: Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor Reinforcement Learning with Deep Energy-...
0 votes
0 answers
41 views

Looking for a reinforcement learning algorithm that deals well with a model-based, curiosity-driven approach for chess AI

I am a software engineer that meddled with machine learning (classifiers) during my thesis. After being out of it for a while I decided I want to try and do a neural network project to learn from, ...
  • 119
0 votes
0 answers
13 views

Is there any benefit to using different polyak numbers for target policy and target Q-function parameters?

In the context of RL and DDPG, it seems polyak averaging is used with reference to two things; the target policy, and the target Q parameters. I know that these hyper parameters can be a bit finicky ...
  • 133
0 votes
0 answers
26 views

A2C unable to solve Cartpole

I have coded my own A2C implementation using PyTorch. However, despite having followed the algorithm pseudo-code from several sources, my implementation is not able to achieve a proper Cartpole ...
  • 16
0 votes
1 answer
39 views

Time index in TD(0) return

in TD(0). for the return we have: $G_{t:t+1}=r_{t+1}+\gamma v_t(s_{t+1})$. Why is the time index on right hand side in $v$ is $t$?
1 vote
0 answers
40 views

Prioritized experience replay correction with off-policy estimators

Prioritized exeperience replay (PER) biases the sampling and introduces importance sampling (IS) correction to the Q-function update. Weights are $w = \frac{1}{N P}^\beta$, where $N$ is the batch size ...
  • 111
1 vote
1 answer
28 views

Confusion in subscript for n-step TD(0)

This is n-step TD(0) update rule: $v_{k+n}(x_k)=v_{k+n-1}(x_k)+\alpha [g_{k:k+n}-v_{k+n-1}(x_k)]$ Why is the subscript on the left hand-side of equation "k+n", not "k+n-1"? Does ...
0 votes
0 answers
25 views

Understanding SARSA with binary features and function approximation

I'm reading the Sutton and Barto Book on Reinforcement Learning (for oral exam preparation). However, I'm stuck on the algorithm for SARSA($\lambda$) with binary features and linear function ...
0 votes
0 answers
25 views

Very high dimensional optimization with large budget, requiring high quality solutions

What would be theoretically the best performing optimization algorithm(s) in this case? Very high dimensional problem: 250-500 parameters Goal is to obtain very high quality solutions, not just "...
0 votes
0 answers
15 views

Dealing with long running tasks in Q Learning

Assuming Q Learning is applied not to low level behaviors (such as taking a step; drawing/playing a card; moving one piece on a board), but rather longer running high-level behaviors (e.g. Moving from ...
  • 1
0 votes
0 answers
22 views

Using AI to Play a Coin Flipping Game?

Imagine I have the following coins: ...
1 vote
0 answers
9 views

Representing non-fixed state-spaces in RL algorithms

How I can define the state of an environment where I don’t have a fixed representation for states? E.g. the state at one step is a list of size 5 and in another state is a list of size 6. What is the ...
0 votes
0 answers
79 views

Why do we need RL in RLHF?

In RLHF, the reward function is a neural network. This means we can compute its gradients cheaply and accurately through backpropagation. Now, we want to find a policy that maximizes reward (see https:...
  • 156
0 votes
1 answer
51 views

What should I do, reinforcement learning agent gives different result on every train?

I'm using PPO+LSTM to create a trading bot. The agent is trained on 3 years of data and tested on 1 year. Every time I train the agent with same set of hyper-parameters, I get very different results ...
1 vote
1 answer
35 views

How to structure the input data for non-vision deep reinforcement learning?

I am currently designing a custom gym environment that is based on sensor data and I struggle a bit with structuring the data and designing the model. Virtually every resource I find online is kind of ...
  • 13
1 vote
1 answer
49 views

How to account for a numeric variable in a state in RL?

I am new to reinforcement learning and having a hard time making the leap from tutorials to real world problems. I'm trying to figure out how to deal with a board game with 9 squares each square ...
  • 13
1 vote
1 answer
86 views

Where is the problem: in batch TD(0) algorithm or in the code to solve AB problem in Sutton-Barto RL book?

Here is the batch TD(0) algorithm: Here is the AB example I want to solve using batch TD(0): And finally here is my Matlab code: % eps1: A 0 B 0 % eps2: B 1 % eps3: B 1 % eps4: B 1 % eps5: B 1 % ...
0 votes
2 answers
61 views

MCTS players keep replaying identical games

I am currently training a self-playing Monte-Carlo-Tree-Search (MCTS) algorithm with a neural network prior, and it seems to be working pretty well. However one problem I have is when I compare my new ...
  • 101
0 votes
1 answer
47 views

Why does $\alpha=1$ mean batch MC Learning?

Here is a part of slide 4 from the link: https://tao.lri.fr/tiki-download_wiki_attachment.php?attId=1683 Why does $\alpha=1$ mean batch MC Learning? I do not see this clearly when I compare with ...
2 votes
0 answers
50 views

Can Q-learning and other RL algorithms solve CNF SAT?

I encountered a question about solving CNF SAT using reinforcement learning: A state is a partial substitution to the variables, and each action is choosing an empty variable and set its value (to <...
  • 121
0 votes
0 answers
21 views

How (if possible at all) the rewards (from the reinforcement learning) can be used to generate the data for the supervised learning?

How (if possible at all) rewards (from reinforcement learning) can be used to generate data for supervised learning? This is very topical question, because human feedback usually comes in the form or ...
  • 803
0 votes
0 answers
25 views

Non-deterministic play in reinforcement learning

I'm training a reinforcement learning agent on simple two-player games (I'm doing Q-learning). I noticed that for a given state the value of each action is usually different. Thus, the greedy policy ...
0 votes
0 answers
29 views

How to increase exploration in IMPALA?

I am using the RLLib library for applying IMPALA algorithm to my environment. I want to know which variables in https://docs.ray.io/en/latest/_modules/ray/rllib/algorithms/impala/impala.html#...
2 votes
1 answer
79 views

Is it always a good idea to use deterministic policies during testing?

I frequently see people setting deterministic = True while testing an RL algorithm. But is this the right approach? For instance, what happens if the agent plays rock, paper, and scissors? In this ...
2 votes
1 answer
27 views

Which Q function do we use to make our policy when using double Q learning?

I know this might be arbitrary, but I couldn't find any good information on this. As we update 2 q function in double q learning I was curios whether we average, or sum them together to get our policy....
  • 23
0 votes
0 answers
24 views

How to train a neural network in a RL environment to learn the optimal strategy in tic tac toe?

I have just been diving into machine learning since last week. I'm trying to create a neural network that learns to play tic tac toe optimally in an RL environment. So far with very little success. ...
  • 101
3 votes
1 answer
111 views

What is the motivation for using Q-Learning in RL?

In Spinning Up by OpenAI, it says the following regarding policy optimization methods and Q-Learning as ways of getting a good policy for RL. Trade-offs Between Policy Optimization and Q-Learning. ...
  • 133
1 vote
1 answer
136 views

Proof of convergence of TD(0) algorithm

I am looking for a proof of the following tabular TD(0) algorithm: However, I can only find proofs with the more general TD($\lambda$) algorithm and I am having problems understanding them. In ...
  • 11

1
2 3 4 5
46