Questions tagged [reinforcement-learning]
For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.
1,848
questions
0
votes
0answers
19 views
How can I use a ResNet as a function approximator for pixel based reinforcement learning?
I'd like to use a residual network to improve learning in image-based reinforcement learning, specifically on Atari Games. My questions contains two parts -
Would it be wise to integrate a generic ...
1
vote
0answers
34 views
Reinforcement Learning method suitable for a large discrete action space with high sample efficiency
Consider the following problem.
We have a process, that generates $N$ stones (e.g. 2000) in one batch $b$. Every pebble has state $s_{i}^b$ and reward $s_i^b$. After choosing one pebble $i$ from the $...
2
votes
0answers
15 views
Demonstration of AI-powered Mario collecting lots of coins?
As part of a talk I'm giving, I'd like to show one of the many videos on YouTube where an AI is playing Mario, such as this one. What bothers me though is that the AI is trying to complete the level ...
3
votes
2answers
85 views
How does $\alpha$ affect the convergence of the TD algorithm?
In Temporal-Difference Learning, we update our value function by $V\left(S_{t}\right) \leftarrow V\left(S_{t}\right)+\alpha\left(R_{t+1}+\gamma V\left(S_{t+1}\right)-V\left(S_{t}\right)\right)$
If we ...
2
votes
0answers
34 views
Why does TD (0) converge to the MLE solution of the Markov model?
Why does TD (0) converge to the MLE solution of the Markov model?
Let's take the Example 6.4 in Sutton and Barto's book as an example.
Example 6.4: You are the Predictor Place yourself now in the ...
2
votes
1answer
36 views
Can RL still learn in a scenario where current state and the next state are independant?
I am trying to implement reinforcement learning into my real-world problem.
One thing making me hesitant to apply RL is that this real-world problem of mine is unique in a way how every state is ...
1
vote
1answer
34 views
Training a reinforcement learning agent that can decide to continue or end the game
I am trying to use reinforcement learning to let an agent learn simultaneously how to play a game and when to end a game.
The task is to find a single target in a grid of locations. At each time step, ...
1
vote
0answers
21 views
Can we learn a policy network via a sequence of manually determined actions?
In policy gradients, is it possible to learn the policy if the chain of actions is selected and performed manually/externally (e.g. by myself or by someone else who I have no influence over)?
For ...
4
votes
2answers
78 views
Is it possible to tell the Reinforcement Learning agent some rules directly without any constraints?
I try to apply RL for a control problem, and I intend to either use Deep Q-Learning or SARSA.
I have two heating storage systems with one heating device, and the RL agent is only allowed to heat up 1 ...
4
votes
2answers
98 views
When to use Value Iteration vs. Policy Iteration
Both value iteration and policy iteration are General Policy Iteration (GPI) algorithms. However, they differ in the mechanics of their updates. Policy Iteration seeks to first find a completed ...
0
votes
0answers
15 views
reinforcement learning with delayed single reward?
I have a case, where my agent has to make actions within a batch of samples, and get a single float reward, such as 1, 0 or -1 as feedback. Each action if of four dimensions. Like a problem of ...
1
vote
0answers
23 views
How should I choose a reinforcement learning algorithm? [closed]
I'm starting a new RL project. I'm familiar with Deep Q-Learning because of an old project where I used it, but I'm not sure I chose correctly back then.
Why should or shouldn't I choose DQN, or any ...
2
votes
1answer
46 views
How to encourage the reinforcement-learning agent to reach the goal as quickly as possible, and what's the effect of discount factor?
I am trying to use reinforcement learning to solve a task and compare its performance to humans.
The task is to find a single target in a fixed number of locations. At each step, the agent will pick ...
1
vote
1answer
22 views
How do I compute the value function when the reward is only at the end in the context of actor-critic algorithms?
Consider the actor-critic reinforcement learning setting (actor and critic parameterized by a neural network). The reward is given only at the end of the episode (or when there is a timeout there is ...
0
votes
0answers
27 views
Should training be done in number of episodes or number of timesteps?
In the past few deep reinforcement learning projects I've worked with, a problem I have encountered is whether to frame training time in terms of number of episodes or number of timesteps. For ...
0
votes
0answers
25 views
How do I solve a minimization problem with Q-learning learning?
I am trying to learn reinforcement learning by myself and so I have a lot of doubts. In particular, I am investigating how to use Q-learning in order to solve minimization problems.
For example, ...
1
vote
1answer
26 views
Do you need a terminal state when using double deep q networks?
I just got my agent training, and I'm wondering if the terminal flags are necessary when sampling from the replay buffer. The game I'm implementing the agent in has two different ways the game can end,...
0
votes
1answer
43 views
Appropriate ML algorithm to solve a cutting pattern problem
I have a rectangular area, where I need to place some 2 dimensional geometrical shapes - like a square or circle or a little more complicated shapes. And after the arrangement these shapes should be ...
0
votes
0answers
25 views
Usefulness of the state_values calculation in Dynamic Programming
State values are always presented as a central concept in RL, notoriously in the bible, the Sutton&Barton’s book.
I have done some exercises trying to improve my understanding, but it is clear ...
1
vote
1answer
44 views
What is the difference between gradient decent in neural networks and temporal difference in reinforcement learning?
I am studying Q-learning in reinforcement learning. My question is about the Bellman equation.
In Q-learning, the Bellman equation is often introduced as follows.
\begin{align}
Q_{new}(s,a)
&= Q_{...
1
vote
0answers
12 views
Book/course recommendation on game theory application to multi-agent system(reinforcement learning)
Is there any great game theory book or course that discusses the application of game theory to modern reinforcement learning or multi-agent systems? Or a classic reference book that can help me get a ...
0
votes
0answers
29 views
Doubt in calculating the optimal costs and value after n steps of a MDP problem
MDP problem - A server requires information from a sensor. The server would like the information to be
fresh. However, there is a cost to querying information from the sensor. Specifically, the state ...
0
votes
1answer
42 views
Can a Reinforcement Learning problem with multiple simultaneous actions be formalized as a Multiagent Partially Observable Markov Decision Process?
Consider the following decision making problem. We have a controller that selects locations from a grid of coordinates and captures an image (observation $o_t$) with a camera at each location (action $...
1
vote
0answers
22 views
Reinforcement Learning for Finite Time Horizon and Non-Trivial Terminal Reward
I notice that most Deep Reinforcement Learning (DRL) works focus on Markov Decision Process (MDP) with an infinite time horizon. Are there any algorithms that work well on finite MDP and non-trivial ...
1
vote
2answers
87 views
How to measure Deep RL algorithms in terms of safety?
I applied for a Ph.D. in AI, my advisor told me that my thesis is about safe applications of deep RL algorithms in healthcare. So I decided to do as the first paper, a comparison of Deep RL algorithms ...
3
votes
1answer
56 views
In reinforcement learning, why are policies defined as functions of states and not observations?
I am new to RL and I am following Sutton & Barto's book.
My doubt is, when we talk about the policy of our agent, we say it is the probability of taking some action $a$ given the state $s$. ...
2
votes
1answer
33 views
Is it possible to apply a particular exploration policy for the on-policy RL agents?
Is it possible to use any particular strategy to explore (e.g. metaheuristics) in on-policy algorithms (e.g. in PPO) or is it only possible to define particular policies to explore in off-policy ...
0
votes
0answers
31 views
What deep reinforcement learning algorithm should I use for my problem?
So here is a description of my problem:
Essentially, I have a large amount of files filled with code for a number of different tasks. However, lets say these codes are inefficient, and should be ...
0
votes
1answer
29 views
Discard irrelavant states from a MDP
I came across this question about MDP.
From the look of it, it seems the full MDP is reducible if the discarded state only have 1 way in and out but is it really so if we change the discounted factor? ...
0
votes
0answers
26 views
Why would the agent always take the same action in the test environment?
I'm trying to set up an agent with PPO2. But I've tried with:
A2C
DQN
PPO2
However, in the test environment, the agent always takes the same action, and the total profit is negative.
What can be the ...
1
vote
0answers
57 views
What is the derivative of equation 1 in the paper "Conservative Q-Learning for Offline Reinforcement Learning"?
I am looking at the paper Conservative Q-Learning for Offline Reinforcement Learning, but I'm not sure how they proved theorem 3.1.
Here is a screenshot of theorem 3.1.
In the proof of theorem 3.1
...
0
votes
0answers
9 views
Why does fictitious self-play use the data collected by the average strategy for reinforcement learning?
I'm reading paper "Fictitious Self-Play in Extensive-Form Games", which introduces fictitious self-play(FPS).
In extensive-form games, let $\beta$ be the best response strategy, $\pi$ be the ...
0
votes
0answers
18 views
Joined vs Separate optimizer for Actor-Critic
Say that I have a simple Actor-Critic architecture, (I am not familiar with Tensorflow, but) in Pytorch we need to specify the parameters when defining an optimizer (SGD, Adam, etc) and therefore we ...
0
votes
0answers
16 views
How does the Markov assumption hold true for episodic task?
The Markov assumption assumes that the current state is sufficient for taking the next action.
Consider an episodic task, where the RL agent is trying to learn to play the game of tic-tac-toe. Here, ...
1
vote
1answer
30 views
How are rewards calculated for episodic tasks like playing chess or tic-tac-toe?
I am new to Reinforcement Learning and trying to understand the concept of reaping rewards during episodic tasks. I think in games like tic-tac-toe, rewards will be in terms of a win or lose. But does ...
1
vote
0answers
29 views
Could we add clipping in the output layer of the actor in DDPG?
I have a doubt about how clipping affects the training of the RL agents.
In particular, I have come across a code for training DDPG agents, the pseudo-code is the following:
...
-1
votes
1answer
49 views
Calculating state-value functions in Markov Decision Process
I am watching David Silver's lectures on RL available on YouTube. My question here is with regard to Lecture 2 (Link to Video). At 1:11:00, I could not understand how he is calculating the state-value ...
1
vote
1answer
37 views
Bellman optimality equation - writing the expression in 2 ways
Okay, I know this question is very basic but I saw two versions of the optimaltiy equation for $V_{*}(s)$ (and probably $Q_{*}(s,a)$). The first one is:
and the second one is :
If following ...
1
vote
0answers
31 views
Setting initial values in DDPG to favor better actions
I'm working on a problem using DDPG.
Is it possible to add some intelligence in the initialization phase, such that the convergence time is improved/shortened and local optima are avoided as much as ...
4
votes
1answer
74 views
How to approach a blackjack-like card game with the possibility of cards being counted?
Consider a single-player card game which shares many characteristics to "unprofessional" (not being played in casino, refer point 2) Blackjack, i.e.:
You're playing against a dealer with ...
0
votes
1answer
45 views
Are the Q-values of DQN bounded at a single timestep?
Consider that we have an agent that has a set of thousands of different actions at each timestep. The reward function in $R:S \rightarrow\{0,1\}$. Let $Q_{t}^\pi(s,a)$ be the estimate from the neural ...
1
vote
2answers
47 views
Should I apply normalization to the observations in deep reinforcement learning?
I am new to DRL and trying to implement my custom environment. I want to know if normalization and regularization techniques are as important in RL as in Deep Learning.
In my custom environment, the ...
1
vote
0answers
27 views
How to parallelize multi-agent DDPG (MADDPG)
I am experimenting with MADDPG algorithm implemented in this repo. Since there were only a few agents (2-3) in the implementation (also in the original paper) steps like parameter updates, action ...
1
vote
1answer
68 views
Can I treat "experience" in reinforcement learning as "training data" in statistical learning?
Statistics is a branch of mathematics that extracts useful information from data. The data is generally called as "training data" in statistical (machine) learning.
Consider the following ...
2
votes
2answers
146 views
What is the optimal score for Tic Tac Toe for a reinforcement learning agent against a random opponent?
I guess this problem is encountered by everyone trying to solve Tic Tac Toe with various flavors of reinforcement learning.
The answer is not "always win" because the random opponent may ...
2
votes
0answers
67 views
REINFORCE differentiation on sum or single value?
I'm currently learning Policy-gradient Methods for RL and encountered REINFORCE algorithm. I learned from this site : https://towardsdatascience.com/policy-gradient-methods-104c783251e0 that the ...
0
votes
2answers
35 views
How to represent "terminate episode" for Knapsack problem with a Pointer Network?
I am currently implementing a Pointer Network to solve a simple Knapsack Problem. However, I am a bit puzzled over the correct (or common, or "best") way to give the agent the option to stop ...
1
vote
1answer
43 views
Delayed state observation or caching action in OpenAI gym. Can it still learn?
I am planning to use OpenAI gym for my experiment in real life.
In my experiment design, by the limits of a real-life scenario, I can only receive the state information or the rewards about 2-3 ...
1
vote
1answer
33 views
PPO when does the update happen?
In many places, it says PPO and Actor-Critic methods in general use TD-updates, but in the loss function for PPO, the Value function loss component uses the difference between output of the value ...
4
votes
1answer
39 views
When would you use Evolutionary Strategies over Step-Based Reinforcement Learning
In Salimans et al, 2016, the authors argue that ES should be considered a competitive alternative to MDP-based RL algorithms like Q-Learning, TRPO.
However, in practice, I notice that more often than ...
