Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [reinforcement-learning]

For questions related to learning controlled by external positive reinforcement or negative feedback signal or both, where learning and use of what has been thus far learned occur concurrently.

2
votes
0answers
14 views

Expected SARSA vs SARSA in “RL: An Introduction”

Sutton and Barto state in the 2018-version of "Reinforcement Learning: An Introduction" in the context of Expected SARSA (p. 133) the following sentences: Expected SARSA is more complex ...
1
vote
1answer
13 views

How do I avoid an agent to tend to terminate in a negative state when time needs to be taken into account?

In an unknown environment, how do I avoid an agent to tend to terminate its trajectory in a negative state when time needs to be taken into account? Suppose the following example to make my question ...
2
votes
1answer
22 views

How is equation 8 derived in the paper “Self-critical sequence training for image captioning”?

In the paper "Self-critical sequence training for image captioning", on page 3, they define the loss function (of the parameters $\theta$) of an image captioning system as the negative expected reward ...
1
vote
0answers
16 views

What is the difference between a non-stationary policy and a state that stores time?

This question is related to What does "stationary" mean in the context of reinforcement learning?, but I have a more specific question to clarify the difference between a non-stationary ...
0
votes
0answers
7 views

How to use a function which compares AI players to create a fitness function of one weights set?

We are given a game between exactly two players, with end result either one of them wins or it is a draw. I have these things already done: A Player(List weights) ...
1
vote
1answer
30 views

How do I apply the value iteration algorithm when there are two goal states?

I am working through the famous RL textbook by Sutton & Barto. Currently, I am on the value iteration chapter. To gain better understanding, I coded up a small example, inspired by this article. ...
2
votes
2answers
66 views

Is Windows a bad choice for DRL?

I'm looking into using PPO implementations like OpenAi's SpinningUp and Baselines. However, I fear that these implementations require packages which are not available for Windows. So I'm wondering if ...
1
vote
2answers
31 views

Modelling gut-feeling/subconscious knowledge of stock market traders

Some (stock market) traders have the ability to produce a high percentage of winning trades (80%+, positive return) over years. I had the chance to look into real money trades of two such traders and ...
2
votes
1answer
31 views

Why does Deep Q Network outputs multiple Q values?

I am learning Deep RL following this tutorial: https://medium.freecodecamp.org/an-introduction-to-deep-q-learning-lets-play-doom-54d02d8017d8 I understand everything but one detail: This image shows ...
1
vote
0answers
38 views

Why is gradient ascent necessary when training Actor Critic agents?

I have read a lot on Actor Critic and I'm not convinced that there is a qualitative difference doing direct gradient updates on the network and slightly adjusting a soft-max output in the direction of ...
1
vote
1answer
30 views

Robot Arm Deep Q Learning Actions

Hello I am new to reinforcement learning and robotics. So far I have an understanding of the concept on 2D world. You can make agent move one step in one direction. However, how do you define movement ...
1
vote
2answers
81 views

Is there more than one Q-matrix update formula?

I asked a question a while ago here and since then I've been solving the issues within my code but I have just one question... This is the formula for updating the Q-Matrix in Q-Learning: $$Q(s_t, ...
2
votes
0answers
18 views

Choosing more than one action in a parameterized policy

I would like to implement a variant of policy iteration that can choose one or more actions in each state. An example would be to heal and move in the game of Doom. Parameterizing the power set of ...
3
votes
1answer
45 views

Should I model my problem as a semi-MDP?

I have a system (like a bank) that people (customers) are entered into the systems by a Poisson process, so the time between the arrival of people (two consecutive customers) will be a random variable....
1
vote
0answers
22 views

How the actor use the output from the critic to make action in actor-critic network?

I am reading about the actor-critic architecture. I am confused about how the actor determines the action using the value (or future reward) from the critic network. Below you have the most popular ...
0
votes
0answers
9 views

How can we estimate the transition model and reward function?

In reinforcement learning (RL), there are model-based and model-free algorithms. In short, model-based algorithms use a transition model (e.g. a probability distribution) and the reward function, even ...
1
vote
2answers
61 views

RL vs Supervised Learning vs Planning

This is an excerpt taken from Sutton and Barto (pg. 3): Another key feature of reinforcement learning is that it explicitly considers the whole problem of a goal-directed agent interacting with ...
0
votes
0answers
25 views

How to approach the sequential decision-making problem with sub-actions with RL?

I have been trying to solve a sequential decision-making problem that involves taking "sub-actions". I would like to share my experience and really appreciate any suggestions/insights. Suppose we ...
0
votes
0answers
16 views

Gridworld Value Function Computation

From example 3.15 of Richard Sutton's book, there's a grid (right) that shows the value function for each state. He mentions this equation being used, along with a uniform random policy. However, I'...
5
votes
4answers
542 views

Is the optimal policy always stochastic if the environment is also stochastic?

Is the optimal policy always stochastic (that is, a map from states to a probability distribution over actions) if the environment is also stochastic? Intuitively, if the environment is ...
2
votes
1answer
20 views

What is the difference between an episode, a trajectory and a rollout?

I often see the terms episode, trajectory and rollout to refer to basically the same thing, a list of (state, action, rewards). Are there any concrete differences between the terms or can they be used ...
1
vote
1answer
23 views

Why is the state value function sufficient to determine the policy if a model is available?

In section "5.2 Monte Carlo Estimation of Action Values" of the second edition of the reinforcement learning book by Sutton and Barto, this is stated: If a model is not available, then it is ...
2
votes
1answer
32 views

how to normalize the state space for articulated robot environments?

What is the common representation used for the state in articulated robot environments? My first guess is that it's a set of the angles of every joint. Is that correct? My question is motivated by the ...
1
vote
1answer
51 views

What are temporal-difference and Monte Carlo methods intuitively?

Intuitively, how do temporal-difference and Monte Carlo methods work in reinforcement learning? How can they be used to solve the reinforcement learning problem?
0
votes
0answers
23 views

How to make episode ending “good” in reinforcement learning?

TL;DR: read the bold. The rest are details I am trying to implement Reinforcement Learning:An Introduction, section 13.5 myself: on OpenAi's cartpole The algorithm seems to be learning something ...
1
vote
0answers
47 views

What are the value functions used in reinforcement learning?

In reinforcement learning, we often define two functions, the "state-value function" $$V^\pi(s) = \mathbb{E}_{\pi}[\sum_{k=0}^{\infty} \gamma^{k}R_{t+k+1}|S_t=s]$$ and the "state-action value ...
0
votes
0answers
12 views

Action exploration using probability matching, hoeffding's inequality (UCB)

At present I use decaying ϵ-greedy to pick what action to take, but I've been looking into ways of better exploring the state-action space. I've read up on hoeffding's inequality / UCB, and ...
0
votes
1answer
87 views

What is the most biologically plausible representation for the actor and critic?

Which representation is most biologically plausible for actor nodes? For example, actions represented across several output nodes which may be either mutually exclusive with each other (e.g., go ...
1
vote
0answers
40 views

Where are reinforcement algorithms used in financial services?

One of the most common misconceptions about reinforcement learning (RL) applications is that, once you deploy them, they continue to learn. And, usually, I'm left having to explain this. As part of my ...
1
vote
1answer
27 views

Are A2C or A3C suitable for episodic tasks where the reward is delivered only at the end of the episode?

My understanding of the main idea behind A2C / A3C is that we run small segments of an episode to estimate the return using a trainable value function to compensate for the unseen final steps of the ...
2
votes
1answer
56 views

Calculating gradient for log policy when variance is not constant

I've noticed that when modelling a continuous action space, the default thing to do is to estimate a mean and a variance where each is parameterized by a neural network or some other model. I also ...
2
votes
1answer
35 views

What is the gradient of the objective function in the Soft Actor-Critic paper?

In the paper "Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor", they define the loss function for the policy network as $$ J_\pi(\phi)=\mathbb E_{...
0
votes
0answers
18 views

How to correctly discount actor critic with experience replay?

In my related question, I asked about the one step actor critic from The Reinforcement Learning Book by Richard Sutton et al, section 13.5: The learning is becoming less significant as the episode ...
1
vote
1answer
25 views

In online one step actor critic, why does the weights update become less significant as the episode progresses?

The Reinforcement Learning Book by Richard Sutton et al, section 13.5 shows an online actor critic algorithm. Why do the weights updates depend on the discount factor via $I$? It seems that the more ...
1
vote
1answer
41 views

How to obtain a formula for loss, when given an iterative update rule in gradient descent?

From the reinforcement learning book section 13.3: Using pytorch, I need to calculate a loss, and then the gradient is calculated internally. How to obtain the loss from equations which are stated ...
3
votes
3answers
142 views

Can the rewards be stochastic when the transition model is deterministic?

Suppose we have a deterministic environment where knowing $s,a$ determines $s'$. Is it possible to get two different rewards $r\neq r'$ in some state $s_{\text{fixed}}$? Assume that $s_{\text{fixed}}$ ...
1
vote
1answer
30 views

Is the next state draw from the joint distribution of the previous state and action?

Suppose $G_t$, the discounted return at time $t$ is defined as: $$ G_t \triangleq R_t+\gamma R_{t+1}+\gamma^{2}R_{t+2} + \cdots = \sum_{j=1}^{\infty} \gamma^{k}R_{t+k}$$ where $R_t$ is the reward at ...
0
votes
0answers
20 views

Is continuous action space really necessary for real world granular robotic control?

Robotic control problems are said to require continuous action spaces. But is that really necessary for granular control with real world robotics? At the end of the day, even human movement is ...
1
vote
0answers
17 views

How to train chat bot on infinite non-stationary data?

I have continual simulated data of million sentences of two simulated persons talking to each other in a room and I want to model one of the persons speech. Now, during this period things in the room ...
1
vote
1answer
34 views

What is the relation between a policy which is the solution to a MDP and a policy like $\epsilon$-greedy?

In the context of reinforcement learning, a policy, $\pi$, is often defined as a function from the space of states, $\mathcal{S}$, to the space of actions, $\mathcal{A}$, that is, $\pi : \mathcal{S} \...
0
votes
0answers
33 views

Which and how reinforcement learning algorithms allow to scale their action and state space?

Till where I could learn, reinforcement learning enables to optimize actions to get more reward. There are a good variety of RL algorithms: model-based and model-free; on-policy and off-policy; ...
0
votes
1answer
38 views

Deep Q-learning is not performing well when there are several enemies

I am playing with a deep Q-learning algorithm in my own environment. The network can perform well as long as there is only one enemy. My agent can perform the following actions: ...
2
votes
1answer
59 views

What is the next state for a two-player board game?

I'm using Q-learning to train an agent to play a board game (e.g. chess, draughts or go). The agent takes an action while in state $S$, but then what is the next state (that is, $S'$)? Is $S'$ now ...
1
vote
1answer
52 views

How do updates in SARSA and Q-learning differ in code?

The update rules for Q-learning and SARSA each are as follows: Q Learning: $$Q(s_t,a_t)←Q(s_t,a_t)+α[r_{t+1}+γ\max_{a'}Q(s_{t+1},a')−Q(s_t,a_t)]$$ SARSA: $$Q(s_t,a_t)←Q(s_t,a_t)+α[r_{t+1}+γQ(s_{t+...
1
vote
1answer
54 views

What is the relation between online learning and on-policy algorithms?

In the context of RL, there is the notion of on-policy and off-policy algorithms. I roughly understand the difference between on-policy and off-policy algorithms. Moreover, in RL, there's also the ...
0
votes
0answers
16 views

Should appending zeros to the observation space not affect deep RL algorithms?

In terms of sample complexity, is it just as easy to learn with observation space A as observation A with 10 zero's appended? For example: OpenAI Gym's fetch robotics environment has a state space ...
1
vote
1answer
70 views

How do I convert table-based to neural network-based Q-learning?

I've used a table to represent the Q function, while an agent is being trained to catch the cheese without touching the walls. The first and last row (and column) of the matrix are associated with ...
3
votes
1answer
44 views

Can Q-learning be used to derive a stochastic policy?

In my understanding, Q-learning gives you a deterministic policy. However, can we use some technique to build a meaningful stochastic policy from the learned Q values? I think that simply using a ...
1
vote
2answers
41 views

How are the reward functions $R(s)$, $R(s, a)$ and $R(s, a, s')$ equivalent?

In this video, the lecturer states that $R(s)$, $R(s, a)$ and $R(s, a, s')$ are equivalent representations of the reward function. Intuitively, this is the case, according to the same lecturer, ...
1
vote
1answer
36 views

What should be saved in SARSA prioritized sweeping?

In the book "Reinforcement Learning: An Introduction", by Sutton and Barto, they provided the "Q-learning prioritized sweeping" algorithm, in which the model saves the next state and the immediate ...