Questions tagged [reinforcement-learning]
For questions related to learning controlled by external positive reinforcement or negative feedback signal or both, where learning and use of what has been thus far learned occur concurrently.
580
questions
0
votes
0answers
22 views
Unique game problem (ML, DP, PP etc)
Looking for a solution to my below game problem. I believe it to require some sort of reinforcement learning, dynamic programming, or probabilistic programming solution, but am unsure... This is my ...
1
vote
0answers
19 views
What is ratio of the objective function in the case of continuous action spaces?
I'm trying to implement the proximal policy optimization (PPO) algorithm. I'm confused on how to make it work with continuous action space.
For discrete action space, the output of the network is the ...
2
votes
1answer
22 views
Will the target network, which is less trained than the normal network, output inferior estimates?
I'm having some trouble understanding some parts of the usage of target networks.
I get that having the same network predict the state/action/advantage values for both the current networks can lead ...
3
votes
0answers
40 views
How can I use Q-learning for inventory decision making?
I am trying to model operational decisions in inventory control. The control policy is base stock with a fixed stock level $S$. That is replenishment orders are placed for every demand arrival to take ...
2
votes
0answers
23 views
Will the RL agent implemented as a neural network fine-tune itself?
Normally, when you develop a neural network, train it for object recognition (on normal objects like bike, car, plane, dog, cloud, etc.), and it turns out to perform very well, you would like to fine-...
0
votes
1answer
31 views
Probabilistic action selection in pursuit algorithm
In the Pursuit algorithm (to balance exploration and exploitation), the greedy action has a probability say $p_1$ (updated every episode) of being selected, while the rest have a probability $p_2$ (...
3
votes
1answer
41 views
How does RL based neural architecture search work?
I have read through many of the papers and articles linked in this thread but I haven't been able to find an answer to my question.
I have built some small RL networks and I understand how REINFORCE ...
6
votes
1answer
128 views
How to stay a up-to-date researcher in ML/RL community?
As a student who wants to work on machine learning, I would like to know how it is possible to start my studies and how to follow it to stay up-to-date. For example, I am willing to work on RL and MAB ...
2
votes
1answer
18 views
Using heuristic dense rewards in a sparse problem
If I am training an agent to try and navigate a maze as fast as possible, a simple reward would be something like
\begin{align}
R(\text{terminal}) &= N - \text{time}\ \ , \ \ N \gg \text{...
1
vote
1answer
39 views
Several questions related to UCT and MCTS
In Bandit Based Monte-Carlo Planning, the article where UCT is introduced as a planning algorithm, there is an algorithm description in page 285 (4 of the pdf).
Comparing this implementation of UCT (...
2
votes
2answers
28 views
Can non-Markov environments also be deterministic?
The definition of deterministic environment I am familiar with goes as follows:
The next state of the agent depends only on the current state and the action chosen by the agent.
By exclusion, ...
2
votes
1answer
43 views
Can one agent command another agent in a multi-agent reinforcement learning setting?
In reinforcement learning, an agent is usually fully autonomous and independent. It executes actions on the environment, but no other agent can control, explore or command this agent.
In multi-agent ...
1
vote
0answers
29 views
Can multiple reinforcement algorithms be applied to the same system?
Can a system, for instance robotic vehicle, be controlled by more than one reinforcement learning algorithm. I intend to use one to address collision avoidance whereas the other to tackle autonomous ...
1
vote
0answers
13 views
Can the agent of reinforcement learning system serve as the environment for other agents and expose actions as services?
Can the agent of reinforcement learning system serve as the environemnt for other agents and expose actions as services? Are there research that consider such question?
I tried to formulate the ...
1
vote
2answers
36 views
How to enforce covariance-matrix output as part of the last layer of a Policy Network?
I have a continuous state space, and a continuous action space. The way I understand it, I can build a policy network which takes as input a continuous state vector and outputs both mean vector and ...
2
votes
0answers
57 views
How does Friend-or-Foe Q-learning intuitively work?
I read about Q-Learning and was reading about multi-agent environments. I tried to read the paper Friend-or-Foe Q-learning, but could not understand anything, except for a very vague idea.
What does ...
1
vote
0answers
57 views
How do I determine the generalisation ability of a neural network?
I am trying to ascertain if my neural network is able to generalize or if it’s simply using memory/overfitting to solve a task. I would like my model to generalise.
Currently, I train the neural ...
1
vote
1answer
20 views
Understanding TD(0) algorithm implementation
I came across the $TD(0)$ algorithm from Sutton and Barto:
Clearly, the only difference of TD methods with the MC methods is that TD method is not waiting till the end of the episode to update the $...
0
votes
0answers
16 views
Why epsilon-greedy hyperparameter is annealed smoothly?
Regarding of DQN, or DQRNN, (reinforcement learning)
To me, RL is a process that can be divided into 2 stages:
Exploring wide range of paths (acting randomly)
Refining the current optimal paths (...
1
vote
1answer
41 views
Why does GLIE+MC Control Algorithm use a single episode of Monte Carlo evaluation?
GLIE+MC control Algorithm:
My question is why does this algorithm use only a single Monte Carlo episode (during PE step) to compute the $Q(s,a)$? In my understanding this has the following drawbacks:
...
0
votes
0answers
33 views
What is a non-stationary mean in the context of RL? [duplicate]
I have come across the expression non-stationary mean in the RL lectures by David Silver, and I really could not understand this expression and its difference from a normal mean. So what exactly is ...
1
vote
0answers
51 views
How can I develop this ML/AI system that I want to use in my new mobile app?
I have an idea for a new mobile app. Here is what I want to accomplish using AI;
I want to get an image (png format), (maybe just byte data too), from my application (I'm developing with Unity3D/C#), ...
1
vote
1answer
52 views
What are options in reinforcement learning?
According to a lecture about Reinforcement Learning, the concept of options allows searching the state space of an agent much faster. The lecture came from Nptel [1] (National Program on Technology ...
1
vote
2answers
137 views
Number of states in taxi environment (Dietterich 2000)
Dietterich, who introduced the taxi environment (see p. 9), states the following: In total there “are 500 [distinct] possible states: 25 squares, 5 locations for the passenger (counting the four ...
0
votes
1answer
47 views
Importance of initialisation of State-Action/State values in RL
I was wondering is there any empirical/theoretical evidence on the effect of initial values of State-Action/State values on the training of an RL agent (the values an RL agent assigns to visited ...
0
votes
0answers
29 views
Questions performance SimPLe pong for AI demo
For a demo I need to develop an AI solution to learn how to play pong. I have the following requirements:
Computer needs to play against a human player. Learn while playing the game.
Poor AI result ...
1
vote
0answers
47 views
Doubt regarding improvement of State Value by n-step returns
Excerpt from Sutton and Barto:
The expected value of all $n$-step returns is guaranteed to improve in a
certain way over the current value function as an approximation to the
true value ...
0
votes
1answer
45 views
Is DDPG just for deterministic environments?
I want to develop an AI for continuous space. I reached to DDPG algorithm that takes actions deterministically.
If DDPG takes actions deterministically, should the environment also be deterministic? ...
6
votes
1answer
115 views
Does AlphaZero use Q-Learning?
I was reading the AlphaZero paper Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm, and it seems they don't mention Q-Learning anywhere.
So does AZ use Q-...
3
votes
1answer
24 views
DQN in stochastic environment
I'm trying to apply a DQN to a stochastic environment but i'm having trouble getting it to converge.
I found some similar questions asked here, but no solutions yet.
I can fairly easy get the DQN to ...
2
votes
1answer
34 views
What is the difference between return and expected return?
At a time step $t$, for a state $S_{t}$, the return is defined as the discounted cumulative reward from that time step $t$.
If an agent is following a policy (which in itself is a probability ...
2
votes
0answers
17 views
Does coarse coding with radial basis function generate fewer features?
I am learning about discretization of state space when applying reinforcement learning to continuous state space. In this video the instructor at 2:02 the instructor says that one benefit of this ...
2
votes
2answers
41 views
How to stop evaluation phase in reinforcement learning with epsilon-greedy Monte Carlo agent?
I have implemented an epsilon-greedy Monte Carlo reinforcement learning agent like suggested in Sutton and Barto's RL book (page 101). As far as I understood epsilon-greedy agents so far, the ...
2
votes
2answers
83 views
Is there any difference between a control and an action in reinforcement learning?
There are reinforcement learning papers (e.g. Metacontrol for Adaptive Imagination-Based Optimization) that use (apparently, interchangeably) the term control or action to refer to the effect of the ...
0
votes
0answers
22 views
What is the difference between a stationary and a non-stationary policy?
In reinforcement learning, there are deterministic and non-deterministic (or stochastic) policies, but there are also stationary and non-stationary policies.
What is the difference between a ...
2
votes
0answers
35 views
Designing state representation for board game
I am trying to write self-play RL (NN + MCTS http://web.stanford.edu/~surag/posts/alphazero.html) to "solve" a board game. However, I got stuck in designing boardgame same (input layer for NN).
1) ...
-2
votes
1answer
44 views
Problem over DQN Algorithm not converging on snake
I'm using a DQN Algorithm to play Snake.
The input of the neural network is a stack of 4 images taken from the games 80x80.
The output is an array of 4 values, one for every direction.
The problem ...
1
vote
1answer
61 views
Reinforcement Learning State Definition
I am quite new to Deep Reinforcement Learning, and I'm trying to define states in a Reinforcement Learning problem. The environment consists of multiple identical elements, and each one of them is ...
2
votes
2answers
78 views
Is there any research work that attempts to combine neuroevolution with deep reinforcement learning
Neuroevolution can be used to evolve a network's architecture (and weights, of course). Deep reinforcement learning, on the other hand, has been proven to be extremely powerful at optimising the ...
0
votes
1answer
33 views
Evolving Machine Learning
It seems to me that, right now, the key to making a good Machine Learning model is in choosing the right combination of hyper-parameters.
Firstly: Am I right in saying, if a model is able to tune it'...
3
votes
1answer
60 views
Knowledge required for understanding AlphaZero paper
My goal is to understand AlphaZero paper published by deepmind. I'm beginning my journey trying to get the basic intuition of reinforcement learning from the book by Barto and Sutton.
As per my ...
0
votes
1answer
44 views
Deep Reinforcement Learning: Rewards suddenly dip down
I am working on a deep reinforcement learning problem. The policy network has the same architecture as the one Deepmind published in 'Playing Atari with Deep Reinforcement Learning'. I am also using ...
4
votes
1answer
49 views
What is the credit assignment problem?
In reinforcement learning (RL), the credit assignment problem (CAP) seems to be an important problem. What is the CAP? Why is it relevant to RL?
1
vote
0answers
21 views
How do I know if the assumption of a static environment is made?
An important property of a reinforcement learning problem is whether the environment of the agent is static, which means that nothing changes if the agent remains inactive. Different learning methods ...
4
votes
1answer
27 views
Why are state transitions in MDPs probabilistic rather than deterministic?
I've read that for MDPs the state transition function $P_a(s, s')$ is a probability. This seems strange to me for modeling because most environments (like video games) are deterministic.
Now, I'd ...
0
votes
0answers
16 views
Weak gradient around one hot values of softmax
If you do the math for the softmax gradient, the gradient is very weak around the simplexical vertices. Aka for a 5-class softmax,[1, 0, 0, 0, 0] has a hard time moving to [0, 1, 0, 0, 0]. I have this ...
0
votes
1answer
63 views
Is Reinforcement Learning possible where no model of the future exists?
Does a form of reinforcement learning exist where an agent can only receive reward based on its current state, rather than a perceived future reward assessed by reasoning over the agent's possible ...
1
vote
1answer
64 views
Why are we using all hyperparameters in RL?
I am new in RL and I am trying to understand why do we need all these hyperparameters.
Can somebody explain me why we use them and what are the best values to use for them?
total_episodes = 50000 ...
3
votes
1answer
77 views
Reinforcement Learning without state space
I want to use Reinforcement Learning to optimize the distribution of energy for a peak shaving problem given by a thermodynamical simulation. However, I am not sure how to proceed as the action space ...
1
vote
0answers
38 views
Can next state and action be same in Deep Deterministic Policy Gradient?
I am trying to apply deep deterministic policy gradient (DDPG) on a robotic application. My states consist of the joint angle positions of the robot and my actions are also its joint angle positions. ...
