Questions tagged [reinforcement-learning]
For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.
1,485
questions
0
votes
0answers
5 views
Hierarchical reinforcement learning for combinatorial complexity
I want to try a hierarchical reinforcement learning (HRL) approach to hard logical problems with combinatorial complexity, i.e. games like chess or Rubik's cube. The majority of HRL papers I have ...
2
votes
0answers
12 views
How are afterstate value functions mathematically defined?
In this answer, afterstate value functions are mentioned, and that temporal-difference (TD) and Monte Carlo (MC) methods can also use these value functions. Mathematically, how are these value ...
1
vote
1answer
62 views
Is the expected value we sample in TD-learning action-value Q or state-value V?
Both MC and TD are model-free and they both follow a sample trajectory (in the case of TD, the trajectory is cut-short) to estimate the return (we basically are sampling Q values). Other than that, ...
0
votes
0answers
14 views
Why does PPO lead to a worse performance than TRPO in the same task?
I am training an agent with an Actor-Critic network and update it with TRPO so far. Now I tried out PPO and the results are drastically different and bad. I only changed from TRPO to PPO, the rest of ...
1
vote
1answer
27 views
Why are shallow networks so prevalent in RL?
In deep learning, using more layers in a neural network adds the capacity to capture more features. In most RL papers, their experiments use a 2 layer neural network. Learning to Reset, Constrained ...
1
vote
0answers
23 views
How exactly is hindsight experience replay related to potential-based reward shaping?
One of the reviewers of the HER paper (which was accepted as a NIPS conference paper) wrote
Overall, I'd say that it's not a huge/deep idea, but a very nice addition to the learning toolbox. When it ...
0
votes
0answers
28 views
DQN fails to learn useful policy for the Taxi environment (Dietterich 200)
I'm building an agent to solve the Taxi environment. I've seen this problem solved with Q-Learning algorithms but my DQN consistently fails to learn anything. The environment has a discrete ...
1
vote
1answer
17 views
Predict next event based on previous events and discrete reward values
Suppose, I have several sequences that include a series of text (the length of sequence can be varied). Also, I have some related reward value. however, the value is not continuous like the text. It ...
1
vote
0answers
22 views
How should I change the hyper-parameters of the C51 algorithm, in order to obtain higher reward?
I have a scenario where, in an ideal situation, the greedy approach is the best, but when non-idealities are introduced which can be learned, DQN starts doing better. So, after checking what DQN ...
2
votes
0answers
24 views
Why is TD(0) not converging to the optimal policy?
I am trying to implement the basic RL algorithms to learn on this 10x10 GridWorld (from REINFORCEJS by Kaparthy).
Currently I am stuck at TD(0). No matter how many episodes I run, when I am updating ...
1
vote
0answers
29 views
How to use unmodified input in neural network?
My question may be a bit hard to explain...
My neural network learns a categorical distribution, which serves as an index. This index will look up the value (= action_mean) in Input 2.
From this ...
1
vote
1answer
44 views
What kind of problems is DQN algorithm good and bad for?
I know this is a general question, but I'm just looking for intuition. What are the characteristics of problems (in terms of state-space, action-space, environment, or anything else you can think of) ...
1
vote
1answer
71 views
Why does Monte Carlo policy evaluation relies on action-value function rather than state-value function?
Here is David Silver's lecture on that. Look at 9:30 to 10:30.
He says that, since it is model-free learning, the environment's dynamics are unknown, so the action-value function $Q$ is used.
But ...
1
vote
0answers
60 views
Does the order in which the features are concatenated to create the state (or observation) matter?
I'm experimenting with an RL agent that interacts with the following environment. The learning algorithm is double DQN. The neural network represents the function from state to action. It's build with ...
4
votes
2answers
271 views
Given two optimal policies, is an affine combination of them also optimal?
If there are two different optimal policies $\pi_1, \pi_2$ in a reinforcement learning task, will the linear combination (or affine combination) of the two policies $\alpha \pi_1 + \beta \pi_2, \alpha ...
2
votes
0answers
33 views
Can $Q$-learning or SARSA be thought of a Markov Chain?
I might just be overthinking a very simple question but nonetheless the following has been bugging me a lot.
Given an MDP with non-trivial state and action sets, we can implement the SARSA algorithm ...
0
votes
0answers
19 views
Is there a resource that explains which settings mean 'High' or 'Low' difficulty in the ALE environment?
I have been using AIgym to train my RL agents. I am now trying to take advantage of the different difficulty settings that the ALE offers.
However I can't find a resource that explains which ...
2
votes
0answers
16 views
Understanding neural network achitectures in policy gradient reinforcement learning for continuous state and action space
I am trying to train a neural network using reinforcement learning / policy gradient methods. The states, i.e. the inputs, as well as the actions I am trying to sample are vectors with each element ...
2
votes
1answer
53 views
When learning off-policy with multi-step returns, why do we use the current behaviour policy in importance sampling?
When learning off-policy with multi-step returns, we want to update the value of $Q(s_1, a_1)$ using rewards from the trajectory $\tau = (s_1, a_1, r_1, s_2, a_2, r_2, ..., s_n, a_n, r_n, s_n+1)$. We ...
0
votes
0answers
21 views
Thompson sampling with Bernoulli prior and non-binary reward update
I am solving a problem for which I have to select the best possible servers (level 1) to hit for a given data. These servers (level 1) in turn hit some other servers (level 2) to complete the request. ...
2
votes
0answers
20 views
How can I discourage the RL agent from drawing in a zero-sum game?
My agent receives $1, 0, -1$ rewards for winning, drawing, and losing the game, respectively. What would be the consequences of setting reward to $-1$ for draws? Would that encourage the agent to win ...
2
votes
0answers
22 views
What is the appropriate way of passing a list of integers that represents the environment to a neural network's dense layer?
I'm training an RL agent using the DQN algorithm to do a specific task. The environment is represented by a list of $10$ integer numbers from $0$ to $20$. An example would be $[5, 15, 8, 8, 0, \dots]$....
1
vote
1answer
40 views
Why aren't exploration techniques, such as UCB or Thompson sampling, used in full RL problems?
Why aren't exploration techniques, such as UCB or Thompson sampling, typically used in bandit problems, used in full RL problems?
Monte Carlo Tree Search may use the above-mentioned methods in its ...
2
votes
1answer
35 views
Can we use Q-learning update for policy evaluation (not control)?
For policy evaluation purposes, can we use the Q-learning algorithm even though, technically, it is meant for control?
Maybe like this:
Have the policy to be evaluated as the behaviour policy.
Update ...
2
votes
0answers
23 views
Is Deep SARSA learning a feasible approach?
I noticed that SARSA has been rarely used in the deep RL setting. Usually, the training for DQN is done off-policy. I think one of the major reasons for this is due to greater sample efficiency in ...
1
vote
1answer
36 views
In reinforcement learning, is it possible to make some actions more likely?
In a general DQN framework, if I have an idea of some actions being better than some other actions, is it possible to make the agent select the better actions more often ?
0
votes
0answers
18 views
OpenAI ES Taylor Second Order Derivative estimation for Newton–Raphson method
I am trying to estimate the second order derivate with vectorial form of taylor expansion for using Newton-Raphson method for some quadratic function optimization. I attached both first order and ...
1
vote
0answers
18 views
What adapts an algorithm to continuous or to discrete action spaces?
Some RL algorithms can only be used for environments with continuous action spaces (e.g TD3, SAC), while others only for discrete action spaces (DQN), and some for both
REINFORCE and other policy ...
0
votes
0answers
23 views
What are the implications of storing the alternative situation (that could have been experienced) in the replay buffer?
Consider an environment where there are 2 outcomes (e.g. dead and alive) and a discrete set of actions. For example, a game where the agent has 2 guns $A$ and $B$ to shoot a monster (the monster dies ...
1
vote
1answer
36 views
Initialising DQN with weights from imitation learning rather than policy gradient network
In AlphaGo, the authors initialised a policy gradient network with weights trained from imitation learning. I believe this gives it a very good starting policy for the policy gradient network. the ...
2
votes
1answer
29 views
How is MuZero's second binary plane for chess defined?
From the MuZero paper (Appendix E, page 13):
In chess, 8 planes are used to encode the action. The first one-hot plane encodes which position the piece was moved from. The next two planes encode ...
1
vote
1answer
39 views
Why do we minimise the loss between the target Q values and 'local' Q values?
I have a question regarding the loss function of target networks and current (online) networks. I understand the action value function. What I am unsure about is why we seek to minimise the loss ...
3
votes
0answers
57 views
In Soft Actor Critic, why is the action sampled from current policy instead of replay buffer on value function update?
While reading the original paper of Soft Actor Critic, I came across on page number 5, under equation (5) and (6)
$$
J_{V}(\psi)=\mathbb{E}_{\mathbf{s}_{t} \sim \mathcal{D}}\left[\frac{1}{2}\left(V_{\...
0
votes
0answers
16 views
Need suggestion for Reinforcement Learning based visual landing system for quadcopters (UAVs)
I have deep interest in quadcopters. I need ideas for designing of an experiment. I have a programmable quadcopter. I can autonomously land it on a staitonary landing pad with a vision algorithm. ...
1
vote
0answers
38 views
Where can I find pre-trained agents able to play games with multiple stages like exploration, dialog, combat?
My goal is to create an ML model to be able to classify different game stages, e.g., dialog with a non-player character, exploration, combat with enemy, in-game menu etc.
In order to do that, I am ...
1
vote
0answers
23 views
Multi-armed bandits: reducing stochastic multi-armed bandits to bernoulli bandits
Agrawal and Goyal (http://proceedings.mlr.press/v23/agrawal12/agrawal12.pdf page 3) discussed how we can extend Thompson sampling for bernoulli bandits to Thompson sampling for stochastic bandits in ...
0
votes
0answers
41 views
Off-policy full-random training in easy-to-explore environment
Let say we are in an environment where a random agent can easily explore all the states of an environment (for example: tic-tac-toe).
In those environments, using off-policy algorithm, is it a good ...
2
votes
1answer
42 views
Is it possible to retrieve the optimal policy from the state value function?
One can easily retrieve the optimal policy from the action value function but how about obtaining it from the state value function?
1
vote
1answer
38 views
Is which sense was AlphaGo “just given a rule book”?
I was told that AlphaGo (or some related program) was not explicitly taught even the rules of Go -- if it was "just given the rulebook", what does this mean? Literally, a book written in ...
0
votes
1answer
33 views
How can reinforcement learning be applied when the goal location or environment is unknown?
I am studying RL. I was thinking whether a new state value or the observation is provided by the environment before the agent actually implements the action.
Take the maze problem as an example. Each ...
2
votes
1answer
28 views
Intuitively, how does it make sense to take an action $A'$ when the environment already ended? [duplicate]
The update equation for SARSA is $Q(S,A) = R + \gamma Q(S',A')$. Consider this: I take an action $A$ that leads to the terminal state. Now my $S'$ would be one of the terminal states. So...
...
0
votes
0answers
41 views
End to End Reinforcement Learning without Reward
I have the following scenario:
Agent: ANN outputs a binary vector $A_t= [a_1, a_2, ..a_n]$
Environment: Outputs States and Rewards, in which:
Each state $S$ is derived from the rewarding function. ...
0
votes
0answers
14 views
Is there a way to import custom Reinforcement Learning Models into Unity? [migrated]
Unity provides two RL algorithms to train agents: PPO and SAC.
I have been searching for weeks now on how to write my own algorithms and only found a mention of a gym-unity wrapper that wraps Unity ...
1
vote
1answer
50 views
How do I create a custom gym environment based on an image?
I am trying to create my own gym environment for the A3C algorithm (one implementation is here). The custom environment is a simple login form for any site. I want to create an environment from an ...
0
votes
0answers
34 views
Is using Bellman Optimality Equation to evaluate states a bad idea when episode number is low?
I am trying to build an RL agent that interacts with an environment, a 2D grid of dimensions 20*10: each (i,j) square in the grid gives out some reward to the agent when it visits that square. Each ...
0
votes
0answers
17 views
Relative Weighting of Loss Weights for Self-Play Reinforcement Learning
I am training some self play reinforcement learning agents to play 2 player board games like Connect 4, Othello, and The Game of the Amazons.
For each game, there is a single neural network with 2 ...
4
votes
1answer
100 views
Why does a negative reward for every step really encourage the agent to reach the goal as quickly as possible?
If we shift the rewards by any constant (which is a type of reward shaping), the optimal state-action value function (and so optimal policy) does not change. The proof of this fact can be found here.
...
0
votes
0answers
14 views
REINFORCE Agent suddenly drops. How to verify if it's due to catastrophic forgetting?
I am using the default implementations of REINFORCE, DQN and c51 available from the tf.agents repo (links). As you can see, DQN manages to improve performance while REINFORCE seems to suffer from ...
1
vote
2answers
71 views
Why not use the target network in DQN as the predictor after training
Target network in DQN is known to make the network more stable, and the loss is like "how good I'm now compared to using the target". What I don't understand is, if the target network is the ...
0
votes
1answer
87 views
What is the optimal value function of the shifted version of the reward function?
Similarly to this question that I asked some time ago, what is the optimal value function of the shifted (by some constant $c$) version of some reward function? More precisely, let's assume that $r(s, ...
