Questions tagged [reinforcement-learning]

For questions related to learning controlled by external positive reinforcement or negative feedback signal or both, where learning and use of what has been thus far learned occur concurrently.

Filter by
Sorted by
Tagged with
0
votes
0answers
15 views

How to compute the target for double Q-learning update step?

I've already read the original paper about double DQN but I do not find a clear and practical explanation of how the target $y$ is computed, so here's how I interpreted the method (let's say I have 3 ...
0
votes
1answer
27 views

Where can I find short videos of examples of RL being used?

I would like to add a short ~1-3 minute video to a presentation, to demonstrate how Reinforcement Learning is used to solve problems. I am thinking something like a short gif of an agent playing an ...
5
votes
2answers
159 views

What are some online courses for deep reinforcement learning?

What are some (good) online courses for deep reinforcement learning? I would like the course to be both programming and theoretical. I really liked David Silver's course, but the course dates from ...
1
vote
0answers
76 views

Comprehensive list of MOOCs and books on Reinforcement Learning [duplicate]

I'm actually trying to learn more about reinforcement learning but I've some trouble to find good resources. Right now I'm in the condition where I'm not so good on the topic to fully understand the ...
3
votes
2answers
92 views

Why are the Bellman operators contractions?

In these slides, it is written \begin{align} \left\|T^{\pi} V-T^{\pi} U\right\|_{\infty} & \leq \gamma\|V-U\|_{\infty} \tag{9} \label{9} \\ \|T V-T U\|_{\infty} & \leq \gamma\|V-U\|_{\infty} \...
3
votes
2answers
92 views

What introductory books to reinforcement learning do you know, and how do they approach this topic?

Currently, I'm only going through these two books Reinforcement Learning: An Introduction, by Sutton and Barto: RL explained on an engineering level (mathematical, but readable for a non-...
4
votes
1answer
41 views

Why are policy iteration and value iteration studied as separate algorithms?

In Sutton and Barto's book about reinforcement learning, policy iteration and value iterations are presented as separate/different algorithms. This is very confusing because policy iteration includes ...
1
vote
1answer
33 views

Is value iteration stopped after one update of each state?

In section 4.4 Value Iteration, the authors write One important special case is when policy evaluation is stopped after just one sweep (one update of each state). This algorithm is called value ...
3
votes
0answers
26 views

What does the number of required expert demonstrations in Imitation Learning depend on?

I just read the following points about the number of required expert demonstrations in imitation learning, and I'd like some clarifications. For the purpose of context, I'll be using a linear reward ...
0
votes
0answers
10 views

Neural network architecture for DDPG agent in Matlab - standard networks?

I want to dive into Reinforcement Learning and therefore as a little project I am trying to swing up and balance a Furuta pendulum in Simulink. Unfortunately I don´t really know where to start, when ...
1
vote
1answer
22 views

Finding the optimal policy from a set of fixed policies in reinforcement learning

This is an open-ended question.Suppose I have a reinforcement learning task that is being solved using many different fixed policies, one of which is optimal. The goal of the agent is not to figure ...
5
votes
1answer
1k views

Can TD($\lambda$) be used with deep reinforcement learning?

TD lambda is a way to interpolate between TD(0) - bootstrapping over a single step, and, TD(max), bootstrapping over the entire episode length, or, Monte Carlo. Reading the link above, I see that an ...
1
vote
1answer
42 views

What is the surrogate loss function in imitation learning, and how is it different from the true cost?

I've been reading A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning lately, and I can't understand what they mean by the surrogate loss function. Some relevant ...
1
vote
1answer
27 views

In DQN, when do the parameters in the Neural Network update based on the reward received?

I'm aware that we back-propagate after computing the loss between: The Neural Network Q values and the Target Network Q values However, all this is doing is updating the parameters of the Neural ...
1
vote
1answer
26 views

Why are Target Networks used in Deep Q-Learning as opposed to the Expected Value equation?

I understand we use a target network because it helps resolve issues regarding stability, however, that's not what I'm here to ask. What I would like to understand is why a target network is used as a ...
2
votes
1answer
21 views

What are the pros and cons of sparse and dense rewards in reinforcement learning?

From what I understand, if the rewards are sparse the agent will have to explore more to get rewards and learn the optimal policy, whereas if the rewards are dense in time, the agent is quickly guided ...
0
votes
0answers
19 views

Customized food for persons based on their profile using Reinforcement learning

I am newbie to Reinforcement Learning, this is my idea - Agent(food provider) has to select a food based on the environment(based on the user profile). Here the reward will be given to the agent based ...
4
votes
2answers
88 views

How can we prevent AGI from doing drugs?

I recently read some introductions to AI alignment, AIXI and decision theory things. As far as I understood, one of the main problems in AI alignment is how to define a utility function well, not ...
1
vote
1answer
42 views

When using experience replay in reinforcement learning, which state is used for training?

I'm slightly confused about the experience replay process. I understand why we use batch processing in reinforcement learning, and from my understanding, a batch of states is input into the neural ...
12
votes
1answer
3k views

What is the Bellman operator in reinforcement learning?

In mathematics, the word operator can refer to several distinct but related concepts. An operator can be defined as a function between two vector spaces, it can be defined as a function where the ...
0
votes
0answers
10 views

DQN in non-episodic tasks?

Are there any reference papers that DQN-like methods are used in continuous, non-episodic tasks?
5
votes
2answers
522 views

Why are lambda returns so rarely used in policy gradients?

I've seen monte-carlo reward $G_{t}$ used in REINFORCE and TD($0$) reward $r_t + \gamma Q(s', a')$ used in vanilla actor-critic. I've never seen someone use lambda reward $G^{\lambda}_{t}$ in these ...
1
vote
0answers
26 views

My Deep Q-Learning Network does not learn for OpenAI gym's cartpole problem

I am implementing OpenAI gym's cartpole problem using Deep Q-Learning (DQN). I followed tutorials (video and otherwise) and learned all about it. I implemented a code for myself and I thought it ...
2
votes
0answers
49 views

Should I use the discounted average reward as objective in a finite-horizon problem?

I am new to reinforcement learning, but, for a finite horizon application problem, I am considering using the average reward instead of the sum of rewards as the objective. Specifically, there are a ...
3
votes
1answer
68 views

What is the purpose of argmax in the PPO algorithm?

I'm kinda new to machine learning and still not too solid on math and particularly calculus. I'm currently trying to implement PPO algorithm as described in the spiningUp website : This line is ...
2
votes
1answer
44 views

Should forecasting with neural networks only be treated as a supervised learning (regression) problem?

I have recently made a work about the application of neural networks to time series forecasting, and I treated this as a supervised learning (regression) problem. I have come across the suggestion of ...
3
votes
1answer
51 views

How does the optimization process in hindsight experience replay exactly work?

I was reading the following research paper Hindsight Experience Replay. This is the paper that introduces a concept called Hindsight Experience Replay (HER), which basically attempts to alleviate the ...
1
vote
0answers
19 views

Trying to proof off policy TD Learning formula

I was reading the book "Introduction to Reinforcement Learning" by Richard Sutton In section 7.3 he write the formula for n-step off-policy TD as:. $$V(S_t) = V(S_{t-1}) + \alpha \rho_{t:t+n-...
1
vote
1answer
27 views

DDPG doesn't converge for MountainCarContinuous-v0 gym environment

I am trying to implement Deep Deterministic policy gradient algorithm by referring to the paper Continuous Control using Deep Reinforcement Learning on the MountainCarContinuous-v0 gym environment. I ...
4
votes
1answer
1k views

Why is GLIE Monte-Carlo control an on-policy control?

In slide 16 of his lecture 5 of the course "Reinforcement Learning", David Silver introduced GLIE Monte-Carlo Control. But why is it an on-policy control? The sampling follows a policy $\pi$ while ...
0
votes
2answers
97 views

What are some (deep) reinforcement learning books for beginners? [duplicate]

What are some books on reinforcement learning (RL) and deep RL for beginners? I'm looking for something as friendly as the head first series, that breaks down every single thing.
0
votes
0answers
21 views

Do TD methods involve $(s,s')$ pairs fitting the Bellman equation on average?

"The basic idea of TD methods are to make state-next state pairs fit the constraints of the Bellman equation on average." Is this statement true? If yes, why, and if not, why not? I'm not ...
2
votes
0answers
14 views

Alternatives to Hierarchical RL for centralized control tasks?

Consider a problem where the agent must learn to control a hierarchy of agents acting against another such agent in a competitive environment. The agents on each team need to learn cooperate in order ...
5
votes
1answer
59 views

Why is the reward in reinforcement learning always a scalar?

I'm reading Reinforcement Learning by Sutton & Barto, and in section 3.2 they state that the reward in a Markov decision process is always a scalar real number. At the same time, I've heard about ...
3
votes
1answer
72 views

When do SARSA and Q-Learning converge to optimal Q values?

Here's another interesting multiple-choice question that puzzles me a bit. In tabular MDPs, if using a decision policy that visits all states an infinite number of times, and in each state, randomly ...
0
votes
0answers
30 views

Monte Carlo Tree search UCB1 for Tic-Tac-Toe help

I am trying to code a MCTS agent for tic tac toe and i have some theoretical questions regarding MCTS. 1)I am using the UCB1 MCTS $UCB(Si)=average value + 2*sqrt(ln(N)/ni)$ . Considering the image ...
0
votes
0answers
15 views

Why is actor's loss the mean of the predicted Q values and not an error in calculating Q values, in the DDPG algorithm?

I am trying to implement the DDPG algorithm. Why is the actor's loss calculated as the negative mean of the model's predicted Q values in the states we are in? Shouldn't it be like the difference of ...
3
votes
2answers
701 views

When is the Markov decision process not adequate for goal-directed learning tasks?

In the book Reinforcement Learning: An Introduction (Sutton and Barto, 2018). The authors ask Exercise 3.2: Is the MDP framework adequate to usefully represent all goal-directed learning tasks? ...
1
vote
2answers
76 views

Policy invariance under affine transformations of the reward function?

In the context of a Markov decision process, this paper says "it is well-known that the optimal policy is invariant to positive affine transformation of the reward function". On the other hand, ...
0
votes
0answers
31 views

How does DQN convergence work in reinforcement learning

In supervised learning we have an unbiased target value, but in reinforcement learning this isn’t the case The network predicts its own target value, now how exactly does it converge if the network ...
2
votes
2answers
988 views

Has anyone been able to solve OpenAI's hardcore bipedal walker with their implementation of DDPG?

As the question suggests, I'm trying to see if I can solve OpenAI's hardcore version of their gym's bipedal walker using OpenAI's DDPG algorithm. Below is a performance graph from my latest attempt, ...
2
votes
1answer
43 views

How should we interpret all the different metrics in reinforcement learning?

I'm trying to train some deep RL agents using policy gradient methods like AC and PPO. While training, I have a ton of different metrics being monitored. I understand that the ultimate goal is to ...
3
votes
1answer
42 views

Why does TD Learning require Markovian domains?

One of my friends and I were discussing the differences between Dynamic Programming, Monte-Carlo, and Temporal Difference (TD) Learning as policy evaluation methods - and we agreed on the fact that ...
1
vote
1answer
32 views

Why are state-values alone not sufficient in determining a policy (without a model)?

"If a model is not available, then it is particularly useful to estimate action values (the values of state-action pairs) rather than state values. With a model, state values alone are sufficient ...
1
vote
1answer
29 views

Why is the optimal policy for an infinite horizon MDP deterministic?

Could someone please help me gain some intuition as to why the optimal policy for a Markov Decision Process in the infinite horizon case (agent acts forever), deterministic? Thank you!
3
votes
1answer
69 views

Comparing the derivation of the Deterministic Policy Gradient Theorem with the standard Policy Gradient Theorem

I would like to understand the difference between the standard policy gradient theorem and the deterministic policy gradient theorem. These two theorem are quite different, although the only ...
7
votes
2answers
105 views

Why state-action value function as an expected value of the return and state value function, does not need to follow policy?

I often see, the state-action value function is expressed as: $q_{\pi}(s,a)=\mathbb{E}_{\pi}[R_{t+1}+\gamma G_{t+1} | S_t=s, A_t = a] = \mathbb{E}[R_{t+1}+\gamma v_{\pi}(s') |S_t = s, A_t =a]$ Why ...
3
votes
2answers
100 views

What are some best practices when trying to design a reward function?

Generally speaking, is there a best-practice procedure to follow when trying to define a reward function for a reinforcement-learning agent? What common pitfalls are there when defining the reward ...
1
vote
0answers
20 views

When past states contain useful information, does A3C perform better than TD3, given that TD3 does not use an LSTM?

I am trying to build an AI that needs to have some information about the past states as well. Therefore, LSTMs are suitable for this. Now, I want to know that for a problem/game like Breakout, where ...
2
votes
1answer
67 views

Can recovering a reward function using IRL lead to better policies compared to reward shaping?

I am working on a research project about the different reward functions being used in the RL domain. I have read up on Inverse Reinforcement Learning (IRL) and Reward Shaping (RS). I would like to ...

1
2 3 4 5
27