Questions tagged [reinforcement-learning]

For questions related to learning controlled by external positive reinforcement or negative feedback signal or both, where learning and use of what has been thus far learned occur concurrently.

Filter by
Sorted by
Tagged with
0
votes
1answer
9 views

How to train reinforcement learning agent from raw pixels

how would you train reinforcement learning agent from raw pixels? For example if you have 3 stacked images to sense motion, than how would you pass them to neural networks to output Q-learning values?...
2
votes
1answer
18 views

What's the right way of building a deep Q-network?

I'm new to RL and to deep q-learning and I have a simple question about the architecture of the neural network to use in an environment with a continous state space a discrete action space. I tought ...
3
votes
2answers
42 views

Is there any difference between reward and return in reinforcement learning?

I am reading Sutton and Barto's book on reinforcement learning. I thought that reward and return were the same things. However, in Section 5.6 of the book, 3rd line, first paragraph, it is written: ...
4
votes
2answers
65 views

Why is $G_{t+1}$ is replaced with $v_*(S_{t+1})$ in the Bellman optimality equation?

In equation 3.17 of Sutton and Barto's book: $$q_*(s, a)=\mathbb{E}[R_{t+1} + \gamma v_*(S_{t+1}) \mid S_t = s, A_t = a]$$ $G_{t+1}$ here have been replaced with $v_*(S_{t+1})$, but no reason has ...
1
vote
1answer
28 views

Why do we also need to normalize the action's values on continuous action spaces?

I was reading here tips & tricks for training in DRL and I noticed the following: always normalize your observation space when you can, i.e., when you know the boundaries normalize your ...
3
votes
4answers
343 views

Reinforcement Learning (RL) how to obtain $p(s',r|s,a)$

I am trying to study the book Reinforcement Learning: An Introduction (Sutton & Barto, 2018). In chapter 3.1 the authors state the following exercise Exercise 3.5 Give a table analogous to that ...
3
votes
1answer
30 views

What is the solution to the exercise 3.11 in the RL book by Sutton and Barto

I'm trying to solve the exercise 3.11 from the second-edition from the Sutton and Barto's book Exercise 3.11 If the current state is $S_t$ , and actions are selected according to a stochastic ...
0
votes
0answers
21 views

How to choose hyperparameters in double DQN?

I'm looking for some indications about the tuning of hyperparameters in building my double DQN. I have a time series problem (with about 2000 observations and no terminal state, I have to max the ...
1
vote
2answers
48 views

Are there any good tutorials about training RL agent from raw pixels using PyTorch?

Is there any good tutorials about training reinforcement learning agent from raw pixels using PyTorch? I don't understand the official PyTorch tutorial. I want to train the agent on the atari ...
1
vote
1answer
21 views

When discounted MAB is useful?

Many of multi-armed bandit algorithms are used when the total reward is the sum of all rewards. However, in RL, the discounted reward is mainly used. Why the discounted reward is not prevailing in MAB ...
2
votes
1answer
44 views

What should the action space for the card game Crib be?

I'm working on creating an environment for a card game, which the agent chooses to discard certain cards in the first phase of the game, and uses the remaining cards to play with. (The game is Crib if ...
4
votes
1answer
76 views

Why not more TD(𝜆) in actor-critic algorithms?

Is there either an empirical or theoretical reason that actor-critic algorithms with eligibility traces have not been more fully explored? I was hoping to find a paper or implementation or both for ...
1
vote
0answers
18 views

Is the TD-residual defined for timesteps $t$ past the length of the episode?

Let $\mathcal{S}$ be the state-space in a reinforcement learning problem where rewards are in $\mathbb{R}$, and let $V:\mathcal{S} \to \mathbb{R}$ be an approximate value function. Following the GAE ...
5
votes
2answers
441 views

Why are lambda returns so rarely used in policy gradients?

I've seen monte-carlo reward $G_{t}$ used in REINFORCE and TD($0$) reward $r_t + \gamma Q(s', a')$ used in vanilla actor-critic. I've never seen someone use lambda reward $G^{\lambda}_{t}$ in these ...
0
votes
0answers
29 views

How is the general return-based off-policy equation derived?

I'm wondering how is the general return-based off-policy equation in Safe and efficient off-policy reinforcement learning derived $$\mathcal{R} Q(x, a):=Q(x, a)+\mathbb{E}_{\mu}\left[\sum_{t \geq 0} \...
5
votes
2answers
415 views

Why am I getting the incorrect value of lambda?

I am trying to solve for $\lambda$ using temporal-difference learning. More specifically, I am trying to figure out what $\lambda$ I need, such that $\text{TD}(\lambda)=\text{TD}(1)$, after one ...
1
vote
0answers
96 views

When do you back-propagate errors through a neural network when using TD($\lambda$)?

I have a neural network that I'm want to use to self-play Connect Four. The neural network receives the board state and is to provide an estimate of the state's value. I would then, for each move, ...
4
votes
1answer
1k views

Can TD($\lambda$) be used with deep reinforcement learning?

TD lambda is a way to interpolate between TD(0) - bootstrapping over a single step, and, TD(max), bootstrapping over the entire episode length, or, Monte Carlo. Reading the link above, I see that an ...
5
votes
1answer
100 views

What is the intuition behind TD($\lambda$)?

I'd like to better understand temporal-difference learning. In particular, I'm wondering if it is prudent to think about TD($\lambda$) as a type of "truncated" Monte Carlo learning?
1
vote
0answers
59 views

How can the target rely on untrained parameters?

I'm trying to understand DQN. I understand where the loss function comes from. I'm just unsure about why the target function works in practice. Given the loss function $$ L_i(\theta_i) = [(y_i - Q(s,a;...
2
votes
1answer
50 views

How is $\Delta$ updated in true online TD($\lambda$)?

In the RL textbook by Sutton & Barto section 7.4, the author talked about the "True online TD($\lambda$)". The figure (7.10 in the book) below shows the algorithm. At the end of each step, $V_{...
0
votes
1answer
32 views

What is correct update when the some indexes are not available?

To update the Q table Q-learning takes the arg max of the Q values - the state, value mappings. For example, in tic tac toe the state XOX OXO -X- contains two ...
1
vote
1answer
38 views

Shouldn't the utility function of two-player zero-sum games be in the range $[-1, 1]$?

In Appendix B of MuZero, they say In two-player zero-sum games the value functions are assumed to be bounded within the $[0, 1]$ interval. I'm confused about the boundary: Shouldn't the value/...
3
votes
1answer
38 views

What happens if the opponent doesn't play optimally in minimax?

I just read an article about the minimax algorithm. When you design the algorithm, you assume that your opponent is a perfect player, i.e. it plays optimally. Let's consider the game of chess. What ...
8
votes
1answer
149 views

Are Q-learning and SARSA the same when action selection is greedy?

I'm currently studying reinforcement learning and I'm having difficulties with question 6.12 in Sutton and Barto's book. Suppose action selection is greedy. Is Q-learning then exactly the same ...
2
votes
1answer
56 views

Reinforcement Learning Continuous Control (DDPG): How to avoid thrashing of issued actions? How to reward smooth output over flittering?

Currently I'm working on a continuous state / continuous action controller. It shall control a certain roll angle of an aircraft by issuing the correct aileron commands (between -1...1 continuous). ...
9
votes
3answers
619 views

Does Monte Carlo tree search qualify as machine learning?

To the best of my understanding, the Monte Carlo tree search (MCTS) algorithm is an alternative to minimax for searching a tree of nodes. It works by choosing a move (generally, the one with the ...
0
votes
0answers
30 views

How can I combine word2vec with tf-idf to have concatenated features?

I want to develop a focused crawler using deep reinforcement learning and a priority queue that will work as the crawler frontier. I reckon using x = (state, action)...
0
votes
0answers
13 views

Is there any advantage to using a non-diagonal covariance matrix for a policy distribution?

For reinforcement learning implementations with a gym.spaces.Box action space, which is the product of $k$ real closed intervals, it is common (actually more like ...
1
vote
1answer
68 views

Can tabular Q-learning converge even if it doesn't explore all state-action pairs?

My understanding of tabular Q-learning is that it essentially builds a dictionary of state-action pairs, so as to maximize the Markovian (i.e., step-wise, history-agnostic?) reward. This incremental ...
3
votes
1answer
72 views

How do I convert an MDP with the reward function in the form $R(s,a,s')$ to and an MDP with a reward function in the form $R(s,a)$?

The AIMA book has an exercise about showing that an MDP with rewards of the form $r(s, a, s')$ can be converted to an MDP with rewards $r(s, a)$, and to an MDP with rewards $r(s)$ with equivalent ...
1
vote
1answer
40 views

What is the difference between reinforcement learning and evolutionary algorithms?

What is the difference between reinforcement learning (RL) and evolutionary algorithms (EA)? For some problems, you could presumably co-evolve two "species" populations using evolutionary algorithms ...
1
vote
0answers
15 views

How to learn how to select a subgraph via reinforcement learning?

I have the following problem. I am given a graph with a lot (>30000) nodes. Nodes are associated with a low (<10)-dimensional feature vector, and edges are associated with a low (<10)-...
3
votes
2answers
115 views

What are some online courses for deep reinforcement learning?

What are some (good) online courses for deep reinforcement learning? I would like the course to be both programming and theoretical. I really liked David Silver's course, but the course dates from ...
2
votes
0answers
20 views

Are the final states not being updated in this $n$-step Q-Learning algorithm?

I am reading this paper and in algorithm 3 they describe an $n$-step Q-Learning algorithm. Below is the pseudo-code. From this pseudo-code, it looks as though the final tuples that they would ...
1
vote
1answer
30 views

What is meant by the rank of the scoring function here?

I've been reading this paper on Knowledge Graph Reasoning for Explainable Recommendation lately, and I don't understand a particular section: Specifically, the scoring function $f((r,e)|u)$ maps ...
1
vote
1answer
53 views

Is there any programming practice website for beginners in Reinforcement Learning [closed]

I am doing an online course on Reinforcement Learning from university of Alberta. It focus too much on theory. I am engineering and I am interested towards applying RL to my applications directly. ...
1
vote
0answers
32 views

Calculating the advantage 'gain' of actions in model-free reinforcement learning

I have a simple question about model-free reinforcement. In a model I'm writing about, I want to know the value 'gain' we'd get for executing an action, relative to the current state. That is, what ...
2
votes
1answer
27 views

Can the agent wait until the end of the episode to determine the reward in SARSA?

From Sutton and Barto's book Reinforcement Learning (Adaptive Computation and Machine Learning series) (p. 99), the following definition for first-visit MC prediction, for estimating $V \sim V_\pi$ is ...
1
vote
1answer
52 views

Is the distribution of state-action pairs from sample based planning accurate for small experience sets?

From the David Silver's lecture 8: Integrating Learning and Planning - based on Sutton and Barto - he talks about using sample-based planning to use our model to take a sample of a state and then use ...
2
votes
1answer
29 views

Adversarial Q Learning should use the same Q Table?

I'm creating a RF Q-Learning agent for a two player fully-observable board game and wondered, if I was to train the Q Table using adversarial training, should I let both 'players' use, and update, the ...
1
vote
1answer
71 views

What is a RAM state in the gym's breakout-ram environment?

I have encountered the gym environment and decided to create AI that plays breakout. Here is the link: https://gym.openai.com/envs/Breakout-ram-v0/. The documentation says that the state is ...
1
vote
0answers
28 views

Actor-Critic implementation not learning

I've implemented a vanilla actor-critic and have run into a wall. My model does not seem to be learning the optimal policy. The red graph below shows its performance in cartpole, where the algorithm ...
1
vote
1answer
93 views

Handle non-existing states in q-learning

I am using Q-learning to solve an engineering problem. The objective is to generate a Q-table associating state to Q-values. I created a State vector ...
0
votes
0answers
39 views

How to create MDP (RL) environment for custom problem?

I am trying to solve the scheduling of resources problems using RL/GA. I am stuck on how to create a custom environment for the problem and actually carry out some tests. I read and implemented Q-...
1
vote
0answers
32 views

How to prevent deep Q-learning algorithms to overfit?

I have recently solved the Cartpole problem using double deep Q-learning. When I saw how the agent was doing, it used to go right every time, never left, and it did similar actions all the time. Did ...
1
vote
1answer
35 views

If deep Q-learning starts to choose only one action, is this a sign that the algorithm diverged?

I'm working on a deep q-learning model in an infinite horizon problem, with a continous state space and 3 possible actions. I'm using a neural network to approximate the action-value function. ...
0
votes
2answers
49 views

RL: What should be the output of the NN for an agent trying to learn how to play a game?

Say the game is tic tac toe. I found two possible output layers: Vector of length 9: each float of the vector represents 1 action (one of the 9 boxes in Tic Tac Toe). The agent will play the ...
2
votes
1answer
195 views

What are the pros and cons of using standard deviation or entropy for exploration in PPO?

When trying to implement my own PPO (Proximal Policy Optimizer), I came across two different implementations : Exploration with std Collect trajectories on $N$ timesteps, by using a policy-centered ...
1
vote
1answer
306 views

How to use the LSTM layer in PPO architecture?

What is the best way of using the LSTM layer in PPO architecture? Should I use them in the first layer of both actor and critic, or use them just before the final layer of these networks? Should I ...

1
2 3 4 5
23