Questions tagged [reinforcement-learning]

For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.

Filter by
Sorted by
Tagged with
1
vote
1answer
60 views

What is the difference between gradient decent in neural networks and temporal difference in reinforcement learning?

I am studying Q-learning in reinforcement learning. My question is about the Bellman equation. In Q-learning, the Bellman equation is often introduced as follows. \begin{align} Q_{new}(s,a) &= Q_{...
1
vote
0answers
22 views

Book/course recommendation on game theory application to multi-agent system (reinforcement learning)

Is there any great game theory book or course that discusses the application of game theory to modern reinforcement learning or multi-agent systems? Or a classic reference book that can help me get a ...
0
votes
0answers
37 views

Doubt in calculating the optimal costs and value after n steps of a MDP problem

MDP problem - A server requires information from a sensor. The server would like the information to be fresh. However, there is a cost to querying information from the sensor. Specifically, the state ...
0
votes
1answer
50 views

Can a Reinforcement Learning problem with multiple simultaneous actions be formalized as a Multiagent Partially Observable Markov Decision Process?

Consider the following decision making problem. We have a controller that selects locations from a grid of coordinates and captures an image (observation $o_t$) with a camera at each location (action $...
1
vote
0answers
36 views

Are there any deep RL algorithms that work well on finite MDPs and non-trivial terminal rewards?

I notice that most Deep Reinforcement Learning (DRL) works focus on Markov Decision Process (MDP) with an infinite time horizon. Are there any algorithms that work well on finite MDP and non-trivial ...
1
vote
2answers
98 views

How to measure Deep RL algorithms in terms of safety?

I applied for a Ph.D. in AI, my advisor told me that my thesis is about safe applications of deep RL algorithms in healthcare. So I decided to do as the first paper, a comparison of Deep RL algorithms ...
3
votes
1answer
64 views

In reinforcement learning, why are policies defined as functions of states and not observations?

I am new to RL and I am following Sutton & Barto's book. My doubt is, when we talk about the policy of our agent, we say it is the probability of taking some action $a$ given the state $s$. ...
2
votes
1answer
40 views

Is it possible to apply a particular exploration policy for the on-policy RL agents?

Is it possible to use any particular strategy to explore (e.g. metaheuristics) in on-policy algorithms (e.g. in PPO) or is it only possible to define particular policies to explore in off-policy ...
0
votes
0answers
33 views

What deep reinforcement learning algorithm should I use for my problem?

So here is a description of my problem: Essentially, I have a large amount of files filled with code for a number of different tasks. However, lets say these codes are inefficient, and should be ...
0
votes
1answer
30 views

Discard irrelavant states from a MDP

I came across this question about MDP. From the look of it, it seems the full MDP is reducible if the discarded state only have 1 way in and out but is it really so if we change the discounted factor? ...
1
vote
1answer
76 views

What is the derivative of equation 1 in the paper "Conservative Q-Learning for Offline Reinforcement Learning"?

I am looking at the paper Conservative Q-Learning for Offline Reinforcement Learning, but I'm not sure how they proved theorem 3.1. Here is a screenshot of theorem 3.1. In the proof of theorem 3.1 ...
0
votes
0answers
15 views

Why does fictitious self-play use the data collected by the average strategy for reinforcement learning?

I'm reading paper "Fictitious Self-Play in Extensive-Form Games", which introduces fictitious self-play(FPS). In extensive-form games, let $\beta$ be the best response strategy, $\pi$ be the ...
0
votes
0answers
26 views

Joined vs Separate optimizer for Actor-Critic

Say that I have a simple Actor-Critic architecture, (I am not familiar with Tensorflow, but) in Pytorch we need to specify the parameters when defining an optimizer (SGD, Adam, etc) and therefore we ...
0
votes
0answers
23 views

How does the Markov assumption hold true for episodic task?

The Markov assumption assumes that the current state is sufficient for taking the next action. Consider an episodic task, where the RL agent is trying to learn to play the game of tic-tac-toe. Here, ...
1
vote
1answer
43 views

How are rewards calculated for episodic tasks like playing chess or tic-tac-toe?

I am new to Reinforcement Learning and trying to understand the concept of reaping rewards during episodic tasks. I think in games like tic-tac-toe, rewards will be in terms of a win or lose. But does ...
1
vote
0answers
38 views

Could we add clipping in the output layer of the actor in DDPG?

I have a doubt about how clipping affects the training of the RL agents. In particular, I have come across a code for training DDPG agents, the pseudo-code is the following: ...
-1
votes
1answer
62 views

Calculating state-value functions in Markov Decision Process

I am watching David Silver's lectures on RL available on YouTube. My question here is with regard to Lecture 2 (Link to Video). At 1:11:00, I could not understand how he is calculating the state-value ...
1
vote
1answer
51 views

How are these two versions of the Bellman optimality equation related?

I saw two versions of the optimality equation for $V_{*}(s)$ and $Q_{*}(s,a)$. The first one is: $$ V_{*}(s)=\max _{a} \sum_{s^{\prime}} P_{s s^{\prime}}^{a}\left(r(s, a)+\gamma V_{*}\left(s^{\prime}\...
1
vote
0answers
32 views

Setting initial values in DDPG to favor better actions

I'm working on a problem using DDPG. Is it possible to add some intelligence in the initialization phase, such that the convergence time is improved/shortened and local optima are avoided as much as ...
4
votes
1answer
86 views

How to approach a blackjack-like card game with the possibility of cards being counted?

Consider a single-player card game which shares many characteristics to "unprofessional" (not being played in casino, refer point 2) Blackjack, i.e.: You're playing against a dealer with ...
0
votes
1answer
49 views

Are the Q-values of DQN bounded at a single timestep?

Consider that we have an agent that has a set of thousands of different actions at each timestep. The reward function in $R:S \rightarrow\{0,1\}$. Let $Q_{t}^\pi(s,a)$ be the estimate from the neural ...
1
vote
1answer
33 views

Why does Q-function training not query the Q-function value at unobserved states?

In the paper Conservative Q-Learning for Offline Reinforcement Learning, it is stated (section 3.1, page 3) that standard Q-function training does not query the Q-function value at unobserved states, ...
1
vote
2answers
165 views

Should I apply normalization to the observations in deep reinforcement learning?

I am new to DRL and trying to implement my custom environment. I want to know if normalization and regularization techniques are as important in RL as in Deep Learning. In my custom environment, the ...
1
vote
0answers
42 views

How to parallelize multi-agent DDPG (MADDPG)

I am experimenting with MADDPG algorithm implemented in this repo. Since there were only a few agents (2-3) in the implementation (also in the original paper) steps like parameter updates, action ...
1
vote
1answer
73 views

Can I treat "experience" in reinforcement learning as "training data" in statistical learning?

Statistics is a branch of mathematics that extracts useful information from data. The data is generally called as "training data" in statistical (machine) learning. Consider the following ...
2
votes
2answers
188 views

What is the optimal score for Tic Tac Toe for a reinforcement learning agent against a random opponent?

I guess this problem is encountered by everyone trying to solve Tic Tac Toe with various flavors of reinforcement learning. The answer is not "always win" because the random opponent may ...
2
votes
0answers
69 views

REINFORCE differentiation on sum or single value?

I'm currently learning Policy-gradient Methods for RL and encountered REINFORCE algorithm. I learned from this site : https://towardsdatascience.com/policy-gradient-methods-104c783251e0 that the ...
0
votes
2answers
40 views

How to represent "terminate episode" for Knapsack problem with a Pointer Network?

I am currently implementing a Pointer Network to solve a simple Knapsack Problem. However, I am a bit puzzled over the correct (or common, or "best") way to give the agent the option to stop ...
1
vote
1answer
73 views

Delayed state observation or caching action in OpenAI gym. Can it still learn?

I am planning to use OpenAI gym for my experiment in real life. In my experiment design, by the limits of a real-life scenario, I can only receive the state information or the rewards about 2-3 ...
1
vote
1answer
74 views

PPO when does the update happen?

In many places, it says PPO and Actor-Critic methods in general use TD-updates, but in the loss function for PPO, the Value function loss component uses the difference between output of the value ...
4
votes
1answer
41 views

When would you use Evolutionary Strategies over Step-Based Reinforcement Learning

In Salimans et al, 2016, the authors argue that ES should be considered a competitive alternative to MDP-based RL algorithms like Q-Learning, TRPO. However, in practice, I notice that more often than ...
1
vote
1answer
164 views

How to Weigt Constraints in A Control Problem with Reinforcement Learning

I have a control problem for a heating device of a building with the goal to minimize the electricity costs for one day under a varying price for electricity in every hour. (more details can be seen ...
0
votes
1answer
45 views

Derive Importance Sampling as Expected Value Notation

I'm new to RL. Recently, I took a course on Coursera. In the Off-policy MC method, I learned the concept of Importance Sampling as follows: where the importance sampling ratio is the ratio of the ...
0
votes
1answer
47 views

Reinforcing Learning when action has no effect on the environment

I am trying to get my head around a problem where the action by the agent can not change the environment. Without going into details, my problem is about error correction in an stochastic environment. ...
0
votes
1answer
81 views

Can future information be included in a control problem with Reinforcement Learning?

I have a control problem for a heating device of a building with the goal to minimize the electricity costs for one day under a varying price for electricity in every hour (more details can be seen ...
3
votes
1answer
117 views

Is it really hard to learn in a stochastic environment?

I understand that a stochastic environment is one that does not always lead you to the desired state by giving a particular action $a$ (But the probability to change to a not desire state is fixed, ...
1
vote
1answer
67 views

When to activate batch normalization and dropout in deep Q-learning?

In the vanilla version of deep Q-learning, there are three places where the Q-network is queried: When exploring. When training: a. When calculating the optimal value of the state reached by an ...
0
votes
1answer
33 views

Policies for which the policy improvement theorem holds

According to Reinforcement Learning (2nd Edition) by Sutton and Barto, the policy improvement theorem states that for any pair of deterministic policies $\pi'$ and $\pi$, if $q_\pi(s,\pi'(s)) \geq v_\...
1
vote
1answer
63 views

What is the intuition behind comparing action values to state values in the policy improvement theorem?

Sutton and Barto, in their book (Reinforcement Learning 2nd Edition) begin the discussion of policy improvement by comparing the action value $q_\pi(s, \pi'(s))$ to the state value $v_\pi(s)$. What is ...
2
votes
1answer
52 views

Why must the value of a state under an optimal policy equal the expected return for the best action from that state?

The Sutton and Barto reinforcement learning textbook states that the value of a state under an optimal policy must equal the expected return for the best action from that state. That is, $$v_*(s) = \...
0
votes
1answer
17 views

How to calculate policy probability ratio in multiple action space

I try to solve a navigation problem with PPO; my actions space have three-part: robot linear velocity that is in [-3,3] range (getting from a tanh activation func) robot linear angular that is in [-...
0
votes
1answer
79 views

Is it possible to solve a linear programming problem using reinforcement learning? (DDPG algorithm)

I'm trying to solve a linear programming problem using reinforcement learning. The linear programming problem is: \begin{array}{ll} \text{maximize}_x & C* x \\ \text{subject to}& A*x \le b\\ ...
0
votes
1answer
51 views

Why don't we bootstrap terminal state in n-step temporal difference prediction update equation?

In the algorithm below, when $\tau + n \geq T$, shouldn't the algorithm bootstrap with the value of the next state? For instance, when $T=5, \tau=3, \& \; n=2$, we don't bootstrap the sample ...
0
votes
0answers
16 views

Why does one-step TD strengthen only the last action of the sequence of actions that led to the high reward, while n-step TD the last n actions?

In the caption of figure 7.4 (p. 147) of Sutton & Barto's book (2nd edition), it's written The one-step method strengthens only the last action of the sequence of actions that led to the high ...
2
votes
1answer
35 views

Is Reinforcement Learning capable of learning complex functions (such as producing a 3d model given an image)?

I want to build an AI that can convert an image of a subject into an anatomically accurate 3D model. To do this, I was thinking of adapting the following code for Deep Deterministic Policy Gradient: ...
0
votes
0answers
20 views

Relationship between standard RL and entropy regularized RL (soft Q learning)

Use the standard RL setting, denote the reward as $r(s,a,s')$, and the optimal Q function as $Q^*(s,a)$, optimal value function $V^*(s)$ and optimal policy $\pi^*(a|s) = \arg \max_a Q^*(s,a)$. In the ...
2
votes
1answer
229 views

Can neural networks have continuous inputs and outputs, or do they have to be discrete?

In general, can ANNs have continuous inputs and outputs, or do they have to be discrete? So, basically, I would like to have a mapping of continuous inputs to continuous outputs. Is this possible? ...
1
vote
0answers
82 views

In what situation would you want to use NEAT over reinforcement learning?

NEAT is an evolutionary algorithm. When would you want to use NEAT over more traditional/common RL algorithms like PPO or SAC etc. What advantage does it give you?
0
votes
1answer
58 views

How to incorporate action information in the state input of a DQN?

I am working on an RL problem that I am trying to solve using a Deep Q-network. The problem concerns choosing drivers to take specific taxi orders. I am familiar with most of the existing works and ...
0
votes
0answers
28 views

DDPG or PPO don't work well with my custom non gym environment

I have a project to control a robot with right and left wheel speeds, and my step time is not constant. Because my outputs are continuous (right wheel speed, left wheel speed, and time step), I try to ...

1 2
3
4 5
38