Questions tagged [reinforcement-learning]

For questions related to learning controlled by external positive reinforcement or negative feedback signal or both, where learning and use of what has been thus far learned occur concurrently.

Filter by
Sorted by
Tagged with
0
votes
1answer
39 views

Why is On-Policy MC/TD Algorithm guaranteed to converge to optimal policy?

Let's say we have a task where the cost depends entirely on the path length to a terminal state, so the goal of an agent would be to take actions to reach terminal state as quickly as possible. Now ...
1
vote
0answers
37 views

Reward problem in A2C with multiple simultaneous discrete actions

I've built an A2C model whose actor's network has two different kinds of discrete actions, so the critic would take state and action (note that critic takes 2 actions because in each timestep we will ...
1
vote
0answers
38 views

Unique game problem (ML, DP, PP etc)

Looking for a solution to my below game problem. I believe it to require some sort of reinforcement learning, dynamic programming, or probabilistic programming solution, but am unsure... This is my ...
2
votes
0answers
96 views

What is ratio of the objective function in the case of continuous action spaces?

I'm trying to implement the proximal policy optimization (PPO) algorithm. I'm confused on how to make it work with continuous action space. For discrete action space, the output of the network is the ...
2
votes
1answer
34 views

Will the target network, which is less trained than the normal network, output inferior estimates?

I'm having some trouble understanding some parts of the usage of target networks. I get that having the same network predict the state/action/advantage values for both the current networks can lead ...
3
votes
0answers
81 views

How can I use Q-learning for inventory decision making?

I am trying to model operational decisions in inventory control. The control policy is base stock with a fixed stock level of $S$. That is replenishment orders are placed for every demand arrival to ...
2
votes
0answers
29 views

Will the RL agent implemented as a neural network fine-tune itself?

Normally, when you develop a neural network, train it for object recognition (on normal objects like bike, car, plane, dog, cloud, etc.), and it turns out to perform very well, you would like to fine-...
0
votes
1answer
50 views

Probabilistic action selection in pursuit algorithm

In the Pursuit algorithm (to balance exploration and exploitation), the greedy action has a probability say $p_1$ (updated every episode) of being selected, while the rest have a probability $p_2$ (...
3
votes
1answer
64 views

How does RL based neural architecture search work?

I have read through many of the papers and articles linked in this thread but I haven't been able to find an answer to my question. I have built some small RL networks and I understand how REINFORCE ...
11
votes
1answer
266 views

How to stay a up-to-date researcher in ML/RL community?

As a student who wants to work on machine learning, I would like to know how it is possible to start my studies and how to follow it to stay up-to-date. For example, I am willing to work on RL and MAB ...
2
votes
1answer
28 views

Using heuristic dense rewards in a sparse problem

If I am training an agent to try and navigate a maze as fast as possible, a simple reward would be something like \begin{align} R(\text{terminal}) &= N - \text{time}\ \ , \ \ N \gg \text{...
3
votes
1answer
102 views

Several questions related to UCT and MCTS

In Bandit Based Monte-Carlo Planning, the article where UCT is introduced as a planning algorithm, there is an algorithm description in page 285 (4 of the pdf). Comparing this implementation of UCT (...
2
votes
2answers
45 views

Can non-Markov environments also be deterministic?

The definition of deterministic environment I am familiar with goes as follows: The next state of the agent depends only on the current state and the action chosen by the agent. By exclusion, ...
3
votes
1answer
48 views

Can one agent command another agent in a multi-agent reinforcement learning setting?

In reinforcement learning, an agent is usually fully autonomous and independent. It executes actions on the environment, but no other agent can control, explore or command this agent. In multi-agent ...
1
vote
0answers
33 views

Can multiple reinforcement algorithms be applied to the same system?

Can a system, for instance robotic vehicle, be controlled by more than one reinforcement learning algorithm. I intend to use one to address collision avoidance whereas the other to tackle autonomous ...
1
vote
0answers
15 views

Can the agent of reinforcement learning system serve as the environment for other agents and expose actions as services?

Can the agent of reinforcement learning system serve as the environemnt for other agents and expose actions as services? Are there research that consider such question? I tried to formulate the ...
1
vote
2answers
44 views

How to enforce covariance-matrix output as part of the last layer of a Policy Network?

I have a continuous state space, and a continuous action space. The way I understand it, I can build a policy network which takes as input a continuous state vector and outputs both mean vector and ...
2
votes
0answers
72 views

How does Friend-or-Foe Q-learning intuitively work?

I read about Q-Learning and was reading about multi-agent environments. I tried to read the paper Friend-or-Foe Q-learning, but could not understand anything, except for a very vague idea. What does ...
1
vote
0answers
62 views

How do I determine the generalisation ability of a neural network?

I am trying to ascertain if my neural network is able to generalize or if it’s simply using memory/overfitting to solve a task. I would like my model to generalise. Currently, I train the neural ...
1
vote
1answer
54 views

Understanding TD(0) algorithm implementation

I came across the $TD(0)$ algorithm from Sutton and Barto: Clearly, the only difference of TD methods with the MC methods is that TD method is not waiting till the end of the episode to update the $...
0
votes
0answers
34 views

Why epsilon-greedy hyperparameter is annealed smoothly?

Regarding of DQN, or DQRNN, (reinforcement learning) To me, RL is a process that can be divided into 2 stages: Exploring wide range of paths (acting randomly) Refining the current optimal paths (...
1
vote
1answer
55 views

Why does GLIE+MC Control Algorithm use a single episode of Monte Carlo evaluation?

GLIE+MC control Algorithm: My question is why does this algorithm use only a single Monte Carlo episode (during PE step) to compute the $Q(s,a)$? In my understanding this has the following drawbacks: ...
1
vote
0answers
59 views

How can I develop this ML/AI system that I want to use in my new mobile app?

I have an idea for a new mobile app. Here is what I want to accomplish using AI; I want to get an image (png format), (maybe just byte data too), from my application (I'm developing with Unity3D/C#), ...
0
votes
1answer
68 views

What are options in reinforcement learning?

According to a lecture about Reinforcement Learning, the concept of options allows searching the state space of an agent much faster. The lecture came from Nptel [1] (National Program on Technology ...
1
vote
2answers
172 views

Number of states in taxi environment (Dietterich 2000)

Dietterich, who introduced the taxi environment (see p. 9), states the following: In total there “are 500 [distinct] possible states: 25 squares, 5 locations for the passenger (counting the four ...
1
vote
1answer
55 views

Importance of initialisation of State-Action/State values in RL

I was wondering is there any empirical/theoretical evidence on the effect of initial values of State-Action/State values on the training of an RL agent (the values an RL agent assigns to visited ...
0
votes
0answers
31 views

Questions performance SimPLe pong for AI demo

For a demo I need to develop an AI solution to learn how to play pong. I have the following requirements: Computer needs to play against a human player. Learn while playing the game. Poor AI result ...
1
vote
0answers
51 views

Doubt regarding improvement of State Value by n-step returns

Excerpt from Sutton and Barto: The expected value of all $n$-step returns is guaranteed to improve in a certain way over the current value function as an approximation to the true value ...
0
votes
1answer
84 views

Is DDPG just for deterministic environments?

I want to develop an AI for continuous space. I reached to DDPG algorithm that takes actions deterministically. If DDPG takes actions deterministically, should the environment also be deterministic? ...
6
votes
1answer
164 views

Does AlphaZero use Q-Learning?

I was reading the AlphaZero paper Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm, and it seems they don't mention Q-Learning anywhere. So does AZ use Q-...
3
votes
1answer
84 views

DQN in stochastic environment

I'm trying to apply a DQN to a stochastic environment but i'm having trouble getting it to converge. I found some similar questions asked here, but no solutions yet. I can fairly easy get the DQN to ...
3
votes
2answers
69 views

What is the difference between return and expected return?

At a time step $t$, for a state $S_{t}$, the return is defined as the discounted cumulative reward from that time step $t$. If an agent is following a policy (which in itself is a probability ...
2
votes
0answers
21 views

Does coarse coding with radial basis function generate fewer features?

I am learning about discretization of state space when applying reinforcement learning to continuous state space. In this video the instructor at 2:02 the instructor says that one benefit of this ...
2
votes
2answers
71 views

How to stop evaluation phase in reinforcement learning with epsilon-greedy Monte Carlo agent?

I have implemented an epsilon-greedy Monte Carlo reinforcement learning agent like suggested in Sutton and Barto's RL book (page 101). As far as I understood epsilon-greedy agents so far, the ...
3
votes
2answers
134 views

Is there any difference between a control and an action in reinforcement learning?

There are reinforcement learning papers (e.g. Metacontrol for Adaptive Imagination-Based Optimization) that use (apparently, interchangeably) the term control or action to refer to the effect of the ...
1
vote
1answer
282 views

What is the difference between a stationary and a non-stationary policy?

In reinforcement learning, there are deterministic and non-deterministic (or stochastic) policies, but there are also stationary and non-stationary policies. What is the difference between a ...
2
votes
0answers
44 views

Designing state representation for board game

I am trying to write self-play RL (NN + MCTS http://web.stanford.edu/~surag/posts/alphazero.html) to "solve" a board game. However, I got stuck in designing boardgame same (input layer for NN). 1) ...
-2
votes
1answer
136 views

Problem over DQN Algorithm not converging on snake

I'm using a DQN Algorithm to play Snake. The input of the neural network is a stack of 4 images taken from the games 80x80. The output is an array of 4 values, one for every direction. The problem ...
1
vote
1answer
103 views

Reinforcement Learning State Definition

I am quite new to Deep Reinforcement Learning, and I'm trying to define states in a Reinforcement Learning problem. The environment consists of multiple identical elements, and each one of them is ...
2
votes
2answers
126 views

Is there any research work that attempts to combine neuroevolution with deep reinforcement learning

Neuroevolution can be used to evolve a network's architecture (and weights, of course). Deep reinforcement learning, on the other hand, has been proven to be extremely powerful at optimising the ...
0
votes
1answer
37 views

Evolving Machine Learning

It seems to me that, right now, the key to making a good Machine Learning model is in choosing the right combination of hyper-parameters. Firstly: Am I right in saying, if a model is able to tune it'...
3
votes
1answer
73 views

Knowledge required for understanding AlphaZero paper

My goal is to understand AlphaZero paper published by deepmind. I'm beginning my journey trying to get the basic intuition of reinforcement learning from the book by Barto and Sutton. As per my ...
1
vote
1answer
88 views

Deep Reinforcement Learning: Rewards suddenly dip down

I am working on a deep reinforcement learning problem. The policy network has the same architecture as the one Deepmind published in 'Playing Atari with Deep Reinforcement Learning'. I am also using ...
6
votes
1answer
925 views

What is the credit assignment problem?

In reinforcement learning (RL), the credit assignment problem (CAP) seems to be an important problem. What is the CAP? Why is it relevant to RL?
1
vote
0answers
28 views

How do I know if the assumption of a static environment is made?

An important property of a reinforcement learning problem is whether the environment of the agent is static, which means that nothing changes if the agent remains inactive. Different learning methods ...
4
votes
1answer
36 views

Why are state transitions in MDPs probabilistic rather than deterministic?

I've read that for MDPs the state transition function $P_a(s, s')$ is a probability. This seems strange to me for modeling because most environments (like video games) are deterministic. Now, I'd ...
0
votes
0answers
24 views

Weak gradient around one hot values of softmax

If you do the math for the softmax gradient, the gradient is very weak around the simplexical vertices. Aka for a 5-class softmax,[1, 0, 0, 0, 0] has a hard time moving to [0, 1, 0, 0, 0]. I have this ...
1
vote
1answer
87 views

Is Reinforcement Learning possible where no model of the future exists?

Does a form of reinforcement learning exist where an agent can only receive reward based on its current state, rather than a perceived future reward assessed by reasoning over the agent's possible ...
1
vote
1answer
86 views

Why are we using all hyperparameters in RL?

I am new in RL and I am trying to understand why do we need all these hyperparameters. Can somebody explain me why we use them and what are the best values to use for them? total_episodes = 50000 ...
3
votes
1answer
87 views

Reinforcement Learning without state space

I want to use Reinforcement Learning to optimize the distribution of energy for a peak shaving problem given by a thermodynamical simulation. However, I am not sure how to proceed as the action space ...