Questions tagged [reinforcement-learning]

For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.

Filter by
Sorted by
Tagged with
0
votes
1answer
24 views

Why is Thompson Sampling considered a part of Reinforcement Learning?

I often see Thompson Sampling in RL literature, however, I am not able to relate it to any of the current RL techniques. How exactly does it fit with RL?
0
votes
0answers
25 views

IQN being outperformed by DDQN. Posible reasons?

I have been getting into RL, and I have a DDQN model that learns how to play the super mario 1-1 world. Then, I tried using the code from the IQN paper to play this game (modified the DQN" part ...
2
votes
1answer
47 views

Is there a mathematical formalism to deal with a missing reward signal?

Typically, a Reinforcement Learning learning problem is formalized as finding an optimal policy for a Markov Decision Process (MDP). In many real-life situations, however, an agent can only get ...
2
votes
0answers
26 views

Reduction of state space of the game Connect Four to apply RL algorithms SARSA and Q-Learning

I would like to implement the reinforcement learning algorithms SARSA and Q-Learning for the board game Connect Four. I am familiar with the algorithms and know about their limitations regarding large ...
0
votes
0answers
33 views

tfp.Distributions.Categorical.sample() is picking the same action everytime after certain iterations

I have written a code for an RL agent such that at each state the model calculates the probabilities of all possible actions and samples one action randomly to proceed further. To acheive this, I have ...
0
votes
0answers
24 views

FrozenLake-v0 not training using REINFORCE

I am implementing a simple REINFORCE (policy gradient) algorithm for openAI's FrozenLake-v0 environment. However, it does not seem to learn anything at all. I have used the same neural architecture ...
0
votes
0answers
29 views

How to properly evaluate RL agents while varying evaluation environments?

I am using Stable-Baselines and my goal is to have agents rank a list of items (or rather, put the relevant items on top), similarly to what learning to rank algorithms (e.g. lambdarank) do as part of ...
0
votes
0answers
40 views

What is the time complexity of DDPG algorithm?

Suppose we have a DDPG algorithm. The actor has N input nodes, two hidden layers with J nodes, and S output nodes. The critic has N+S input nodes, two hidden layers with C nodes, and one output node. ...
4
votes
1answer
125 views

Do we use validation and test sets for training a reinforcement learning agent?

I am pretty new to reinforcement learning and was working with some code for the PPO and DQN algorithms. After looking at the code, I noticed that the authors did not include any code to setup a ...
-1
votes
1answer
47 views

Is it a bad practice to use cumulative rewards in reinforcement learning

I am using a DDPG agent for doing prediction on the position on an asset in a stock trading-like environment. I am using the cumulative reward as the reward for each timestep. Since it is trained over ...
1
vote
0answers
57 views

Can Q-learning be used for my scenario, and how might I do so?

I have already asked 2-3 general questions w.r.t Q learning and now I am asking a scenario specific one. I will try to be concise and understandable. I really really need help. Scenario: I have a ...
4
votes
1answer
56 views

When to use the state value function $V(s)$ and when to use the state-action value function $Q(s, a)$?

I saw the difference between value function $V(s)$ and $Q(s, a)$. But when do I use each one? When I coded in Matlab I only used $Q(s, a)$ directly (as I was thinking of a tabular approach). So, when ...
0
votes
0answers
58 views

What do we actually 'approximate' when dealing with large state spaces in Q-learning?

I realized that my state space is very large in size. I had planned to use tabular Q-learning (Bellman equation to update the $Q(s, a)$ after each action taken). But this 'large space' realization has ...
-2
votes
1answer
42 views

Is it possible to train an RL agent using images?

I have an image which consists of a start and an end point, the journey has some obstacles which have to be avoided. Is it possible to train an RL agent using such images to find the best path ...
2
votes
0answers
35 views

How rewards are playing role in Deep Q Network

I have started working on Reinforcement Learning, specifically DQN. And I have watched some interesting videos on it. However, I have some doubts about how the model works. Let's say we are playing ...
-1
votes
1answer
43 views
2
votes
0answers
29 views

How to fix high variance of the returns on a 2d env?

I'm trying to train an agent on a self-written 2d env, and it just doesn't converge to the solution. It is basically a 2d game where you have to move a small circle around the screen and try to avoid ...
2
votes
1answer
47 views

How to code an $\epsilon$-soft policy for on-policy Monte Carlo control?

I was trying to code the on-policy Monte Carlo control method. The initial policy chosen needs to be an $\epsilon$-soft policy. Can someone tell me how to code an $\epsilon$-soft policy? I know how to ...
4
votes
1answer
50 views

In TD(0) with linear function approximation, why is the gradient of $\hat v(S^{\prime}, \mathbf w)$ wrt parameters $\mathbf w$ not considered?

I am reading these slides. On page 38, the update for the parameters for the linear function approximation of TD(0) is given. I have a doubt regarding this. The cost function (RMSE) is given on page ...
1
vote
1answer
70 views

Is it generally advisable to have a low dimensional action space in Reinforcement Learning?

In supervised or unsupervised learning, it is advised to reduce the dimensionality due to the curse of dimensionality in general. Is this also generally advisable for the action space of reinforcement ...
2
votes
0answers
29 views

Which multi-agent reinforcement learning algorithm can I use when there are two types of agents with different action spaces?

Most of the papers on multi-agent RL (MARL) that I have encountered have multiple agents who have a common action space. In my work, my scenario involves $m$ numbers of a particular agent (say type A) ...
2
votes
1answer
55 views

How does this TD(0) off-policy value update formula work?

The update formula for the TD(0) off-policy learning algorithm is (taken from these slides by D. Silver for lecture 5 of his course) $$ \underbrace{V(S_t)}_{\text{New value}} \leftarrow \underbrace{V(...
0
votes
0answers
32 views

How to deal with Q-learning having low variance in predicted Q-values?

I have a neural network that takes the state (which contains a lot of data), and the possible action (which is very little data), and predicts the Q-value of the action. I am double Q-learning. I've ...
3
votes
1answer
68 views

Can an optimal policy have a value function that has a smaller value for a state than a non-optimal policy?

I'm starting to learn about the Bellman Equation and a question came to my mind. A policy $\pi$ is optimal if the value $v_\pi(s)$ is greater or equal than the value $v_{\pi'}(s)$ for all states $s \...
4
votes
0answers
651 views

Why isn't a target network used for the critic in on-policy actor-critic methods?

Based on my research, I've seen so many on-policy AC approaches that utilise a critic network to estimate the value function $V$. The Bellman equation for the value function is as bellow: $$ V_\pi(s_t)...
1
vote
1answer
54 views

Why would SARSA diverge (but not Expected SARSA or Q-learning)?

In figure 6.3 (shown below) from Reinforcement Learning: An Introduction (second edition) by Sutton and Barto, SARSA is shown to perform worse asymptotically (after 100k episodes) than in the interim (...
3
votes
1answer
71 views

Why exactly was previously believed that the deterministic policy gradient did not exist?

I'm reading the paper Deterministic Policy Gradient Algorithms, David Silver et al. First of all, in the introduction, the author says that It was previously believed that the deterministic policy ...
1
vote
0answers
34 views

Multi-armed Bandit in optimization on graph edges selection

I have the problem, which I described below. I wonder if there exists a class of multi-armed bandit approaches that is related to it. I am working on computer networking optimization. In the simplest ...
0
votes
0answers
41 views

Why does this PPO agent doesn't learn at all?

I am trying to code a Proximal Policy Optimization algorithm in Pytorch by myself for the OpenAI gym preduum-v0 environment. However, I find my learning curve is ...
0
votes
1answer
67 views

How should I define the reward function for a stock trading-like game?

Problem setting Consider a game like trading a stock At each step, the agent can buy / sell a stock. Trade is a pair of ...
1
vote
1answer
96 views

Is there a tutorial for understanding the proof of convergence for TD learning?

I'm reading the article An Analysis of Temporal-Difference Learning with Function Approximation (1997), but the mathematics inside seems overly complicated for me. Answers to some similar questions ...
1
vote
1answer
46 views

How can I use a ResNet as a function approximator for pixel based reinforcement learning?

I'd like to use a residual network to improve learning in image-based reinforcement learning, specifically on Atari Games. My main question is divided into 3 parts. Would it be wise to integrate a ...
1
vote
1answer
86 views

Reinforcement Learning method suitable for a large discrete action space with high sample efficiency

Consider the following problem. We have a process, that generates $N$ stones (e.g. 2000) in one batch $b$. Every pebble has state $s_{i}^b$ and reward $s_i^b$. After choosing one pebble $i$ from the $...
2
votes
0answers
21 views

Demonstration of AI-powered Mario collecting lots of coins?

As part of a talk I'm giving, I'd like to show one of the many videos on YouTube where an AI is playing Mario, such as this one. What bothers me though is that the AI is trying to complete the level ...
2
votes
2answers
124 views

How does $\alpha$ affect the convergence of the TD algorithm?

In Temporal-Difference Learning, we update our value function by $V\left(S_{t}\right) \leftarrow V\left(S_{t}\right)+\alpha\left(R_{t+1}+\gamma V\left(S_{t+1}\right)-V\left(S_{t}\right)\right)$ If we ...
2
votes
0answers
43 views

Why does TD (0) converge to the MLE solution of the Markov model?

Why does TD (0) converge to the MLE solution of the Markov model? Let's take the Example 6.4 in Sutton and Barto's book as an example. Example 6.4: You are the Predictor Place yourself now in the ...
2
votes
1answer
39 views

Can RL still learn in a scenario where current state and the next state are independant?

I am trying to implement reinforcement learning into my real-world problem. One thing making me hesitant to apply RL is that this real-world problem of mine is unique in a way how every state is ...
1
vote
1answer
41 views

Training a reinforcement learning agent that can decide to continue or end the game

I am trying to use reinforcement learning to let an agent learn simultaneously how to play a game and when to end a game. The task is to find a single target in a grid of locations. At each time step, ...
1
vote
0answers
21 views

Can we learn a policy network via a sequence of manually determined actions?

In policy gradients, is it possible to learn the policy if the chain of actions is selected and performed manually/externally (e.g. by myself or by someone else who I have no influence over)? For ...
5
votes
3answers
208 views

Is it possible to tell the Reinforcement Learning agent some rules directly without any constraints?

I try to apply RL for a control problem, and I intend to either use Deep Q-Learning or SARSA. I have two heating storage systems with one heating device, and the RL agent is only allowed to heat up 1 ...
3
votes
2answers
343 views

When to use Value Iteration vs. Policy Iteration

Both value iteration and policy iteration are General Policy Iteration (GPI) algorithms. However, they differ in the mechanics of their updates. Policy Iteration seeks to first find a completed ...
0
votes
0answers
16 views

reinforcement learning with delayed single reward?

I have a case, where my agent has to make actions within a batch of samples, and get a single float reward, such as 1, 0 or -1 as feedback. Each action if of four dimensions. Like a problem of ...
1
vote
0answers
29 views

How should I choose a reinforcement learning algorithm? [closed]

I'm starting a new RL project. I'm familiar with Deep Q-Learning because of an old project where I used it, but I'm not sure I chose correctly back then. Why should or shouldn't I choose DQN, or any ...
2
votes
1answer
55 views

How to encourage the reinforcement-learning agent to reach the goal as quickly as possible, and what's the effect of discount factor?

I am trying to use reinforcement learning to solve a task and compare its performance to humans. The task is to find a single target in a fixed number of locations. At each step, the agent will pick ...
1
vote
1answer
36 views

How do I compute the value function when the reward is only at the end in the context of actor-critic algorithms?

Consider the actor-critic reinforcement learning setting (actor and critic parameterized by a neural network). The reward is given only at the end of the episode (or when there is a timeout there is ...
0
votes
0answers
44 views

Should training be done in number of episodes or number of timesteps?

In the past few deep reinforcement learning projects I've worked with, a problem I have encountered is whether to frame training time in terms of number of episodes or number of timesteps. For ...
0
votes
0answers
29 views

How do I solve a minimization problem with Q-learning learning?

I am trying to learn reinforcement learning by myself and so I have a lot of doubts. In particular, I am investigating how to use Q-learning in order to solve minimization problems. For example, ...
1
vote
1answer
42 views

Do you need a terminal state when using double deep q networks?

I just got my agent training, and I'm wondering if the terminal flags are necessary when sampling from the replay buffer. The game I'm implementing the agent in has two different ways the game can end,...
0
votes
1answer
53 views

Appropriate ML algorithm to solve a cutting pattern problem

I have a rectangular area, where I need to place some 2 dimensional geometrical shapes - like a square or circle or a little more complicated shapes. And after the arrangement these shapes should be ...
0
votes
0answers
26 views

Usefulness of the state_values calculation in Dynamic Programming

State values are always presented as a central concept in RL, notoriously in the bible, the Sutton&Barton’s book. I have done some exercises trying to improve my understanding, but it is clear ...

1
2
3 4 5
38