People who code: we want your input. Take the Survey

Questions tagged [reinforcement-learning]

For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.

Filter by
Sorted by
Tagged with
0
votes
0answers
17 views

Comparison between TD(0) and MC ( or GAE )?

I'm getting started with DRL and have trouble distinguishing TD(0), MC, and GAE; and which scenarios one's better than others. Here is what I understand so far: TD(0): increment learning, can learn ...
0
votes
1answer
33 views

What would happen to an agent trained using Markov Decision Process if the goal node changes?

I was reading up a paper that did routing based on an MDP, and I was wondering because, in routing, there is a sender node and a receiver node, so if the receiver node changes (sending a message to ...
3
votes
0answers
90 views

How to deal with a moving target in the Lunar Lander environment with DDPG?

I have noticed that DDPG does rather well at solving environments with a static target. For example, the default of Lunar Lander, the flags do not change position. So the DDPG model learns how to get ...
-1
votes
1answer
35 views

Exploration for softmax should be binary or continuous softmax?

Maybe it's silly to ask but for random exploration in an RL for choosing discrete action, that in the neural network last layer softmax will be used, what random samples should we provide? binary like ...
1
vote
1answer
29 views

PPO agent for vehicle control does not learn to stop at traffic lights

I have built a custom RL environment with gym, which simulates the RL vehicle and potential vehicles in front of the RL vehicle as well as traffic lights (including ...
2
votes
0answers
32 views

suitable reference for reinforcement learning for beginners [duplicate]

I am looking for a reference book on RL for first time learners, one that is a gentle introduction and not as wordy as Sutton/Barto. I am interested in something similar to Georgia Institute of ...
0
votes
0answers
46 views

Reinforcement learning for rearranging the mobile home screen icon layout: what inputs/states do I need to pass into the algorithm?

I have a problem where I need to rearrange a particular user's mobile home screen icon layout. Let's say that the social media app usage of a user is high compared to other app usage. So I need the ...
2
votes
0answers
40 views

Why isn't RL considered a continual learning strategy itself?

I have read about methods that apply continual learning strategies to reinforcement learning. Since reinforcement learning also learns step by step (i.e., task by task, in a sense) during the training ...
2
votes
0answers
33 views

Is there any reasonable notion of regret for infinite horizon discounted MDPs?

I am thinking about episodic MDPs. Usually, in episodic MDPs, it seems that we have a finite fixed horizon per episode and no discount factor. Then, a very intuitive notion of regret after $T$ ...
3
votes
1answer
79 views

In DQN, would it be cheaper to have $N$ neural networks with a single real-valued output, one for each of the $N$ actions?

In the classical examples of deep q-learning, I often see neural networks in which the input represents the state of the agent, while the output is a tuple with all the values of $Q(s, a)$ predicted ...
0
votes
0answers
33 views

Should the actor and critic share a common feature extraction neural network?

In an environment with image observations, if we use an actor-critic method to find a good policy, commonly, we will use a feature extraction neural network, such as ResNet, to extract the information ...
1
vote
1answer
48 views

How to scale all positive continuous reward?

My RL project has all positive continuous rewards for every step and the goal is to have the maximum cumulative reward (episodic reward). The problem is that the rewards are too close and all between ...
1
vote
0answers
43 views

PPO in continuous control not working

I have PPO agent for discrete action space for LunarLander-v2 env in gym and it works well. However, when i am trying to solve continuous version of the same env - <...
0
votes
0answers
19 views

What are the state-of-the-art learning algorithms for contextual bandits with stochastic rewards

I am building a solution for an environment with stochastic rewards in an online setting. I am wondering what the state of the art is in this setting. Is it $\epsilon$-greedy (with logistic regression)...
1
vote
1answer
18 views

Why is it that the state visitation frequency equals the sum of state visitation frequency from initial time step to the horizon?

In the maximum entropy inverse reinforcement learning paper, Ziebart et al. show that the state visitation frequency $\rho(s)$ of a state $s$ can be computed as $$ \rho_{\pi}(s) = \sum_{t}^{T} P(s_t=s|...
2
votes
1answer
62 views

Difference in UCB performance when scaling the rewards

I notice the following behavior when running experiments with $\epsilon$-greedy and UCB1. If the reward is kept binary (0 or 1) both algorithm's performances are on par with each other. However, if I ...
0
votes
0answers
7 views

Update state representation based on agent action

I am working on a project where I have to train a RL agent which will simulate Loan repayment track of a customer's loan based on his features derived from his credit profile (state vector). I am ...
2
votes
2answers
99 views

Why does Alpha Zero's Neural Network flip the board to be oriented towards the current player?

While reading the AlphaZero paper in preparation to code my own RL algorithm to play Chess decently well, I saw that the "The board is oriented to the perspective of the current player." I ...
1
vote
1answer
48 views

What would be the importance sampling ratio for off-policy TD learning control using Q values?

The off-policy TD learning control using state value function from page 34 of David Silver's RL lecture is: $$ V(S_t) \leftarrow V(S_t) + \alpha \left( \frac{ \pi(A_t|S_t)}{\mu (A_t|S_t)} (R_{t+1} + \...
-1
votes
1answer
25 views

Using states (features) and actions from a heuristic model to estimate the value function of a reinforcement learning agent [closed]

new to RL here. As far as i understood from RL courses, that there is two sides of reinforcement learning. Policy Evaluation, which is the task of knowing the value function for certain policy. and ...
1
vote
0answers
31 views

Is this a supervised or reinforcement learning problem, and which algorithm should I use to solve it?

I have a time series data with a little unusual cost/reward function (I haven't seen it before) The model must predict a $Y$ value for any $X(t)$. The reward is computed as follows. The model will ...
3
votes
1answer
102 views

How does the Alpha Zero's move encoding work?

I am a beginner in AI. I'm trying to train a multi-agent RL algorithm to play chess. One issue that I ran into was representing the action space (legal moves/or honestly just moves in general) ...
0
votes
0answers
50 views

Open AI Taxi - Agent fails to learn an effective policy

I'm trying to solve the openai gym taxi problem (v3) using deep q learning. I've already had some success with the q-table approach, but for the life of me cannot manage to train a NN to learn a ...
0
votes
0answers
31 views

Any RL approaches for this 2D space optimisation problem?

I have a list of rectangles, they are in certain order in 2D at the beginning. The task is to move them to get the boundary (rectangular) of the minimal area. It's OK to push off the dotted border as ...
5
votes
1answer
197 views

Does the policy iteration convergence hold for finite-horizon MDP?

Most RL books (Sutton & Barto, Bertsekas, etc.) talk about policy iteration for infinite-horizon MDPs. Does the policy iteration convergence hold for finite-horizon MDP? If yes, how can we derive ...
0
votes
0answers
20 views

Can I use binary numbers and plain floats in the same neural network input for reinforcement learning?

I'm modeling a system whose configuration can be represented by a binary array ([1,0,0,1] or [0,1,0,0], for example), and the agent can move on a 2D space (thus having 3 DOF), and the action the agent ...
1
vote
0answers
57 views

Relation between discounted MDP and stochastic shortest path problems in RL

I have been reading about discounted MDPs and Stochastic Shortest Path (SSP). I recently came to know (from a friend) that every discounted MDP can be converted to an equivalent SSP but not the other ...
1
vote
1answer
38 views

How to properly resume training of deep Q-learning network?

I'm currently training a deep q-learning network. Due to resource limitations, I am not able to train the model to the desired performance in one go. So what I'm doing now is training the model for a ...
0
votes
0answers
29 views

Double DQN backpropagation of negative final rewards?

My problem is that in my Double DQN model, negative final rewards are not being backpropagated into action Q-values, and so some Q-values are positive, when they should be negative, and hence ...
2
votes
2answers
150 views

With Monte Carlo off-policy learning what do we correct by using importance sampling?

I do not understand the link of importance sampling to Monte Carlo off-policy learning. We estimate a value using sampling on whole episodes, and we take these values to construct the target policy. ...
1
vote
1answer
38 views

Why is the logarithm of the standard deviation used in this implementation of proximal policy optimization?

I am currently writing my bachelor thesis, which is an implementation of proximal policy optimization. Sometimes, I hit a wall because of the gaps in my mathematical knowledge. However, implementing ...
2
votes
2answers
239 views

How to fight with unstability in self play?

I'm working on a neural network that plays some board games like reversi or tic-tac-toe (zero-sum games, two players). I'm trying to have one network topology for all the games - I specifically don't ...
2
votes
1answer
55 views

How to recover the target Q network's weights solely from the snapshots of the primary Q network's weights in DQN?

Suppose that I have a DQN agent, which has two neural networks: one is the primary Q network and the other is the target Q network. In every update, the target Q network is updated with a soft update ...
0
votes
0answers
8 views

Is it possible to use agent coordinates directly as part of the state

I am working on graph optimisation problem using DQN - the graph is represented as an adjacency matrix and an agent moves through this matrix removing edges between nodes (add a 0) or adding edges ...
0
votes
0answers
41 views

PyTorch: How to deal with hidden states of an LSTM?

I have a time series in which each date is correlated with the preview one, and base on that I am trying to predict action 1 and action 2. But the problem is that I am not sure how to deal with the <...
-2
votes
1answer
52 views

Why the optimal Bellman operator of a Q-function can be approximated by a single point

I am currently studying reinforcement learning, especially DQN. In DQN, learning proceeds in such a way as to minimize the norm (least-squares, Huber, etc.) of the optimal Bellman equation and the ...
0
votes
0answers
32 views

Backpropagation in REINFORCE algorithms with Categorical / Multinomial Distribution

From a paper by Williams, I know in general how to backpropagate log-probabilities of chosen actions when applying the REINFORCE weight update rule. However, I was wondering about a case not being ...
0
votes
1answer
59 views

Use Reinforcement Learning instead of genetic algorithm for optimization

I want to use RL instead of genetic or any other evolutionary algorithm in order to find the best parameter for a function. Here is the problem: Given a function $$f(x,y,z,data)$$ x,y and z are some ...
6
votes
1answer
192 views

What is ergodicity in a Markov Decision Process (MDP)?

I have read about the concept of ergodicity on the safe RL paper by Moldovan (section 3.2) and the RL book by Sutton (chapter 10.3, 2nd paragraph). The first one says that "a belief over MDPs is ...
4
votes
1answer
43 views

How does a model based agent learn the model?

I want to build model-based RL. I am wondering about the process of building the model. If I already have data, from real experience: $S_1, a \rightarrow R,S_2$ $S_2, a \rightarrow R,S_3$ Can I use ...
0
votes
1answer
54 views

What is the difference between environment states and agent states in terms of Markov property?

I'm going through the David Silver RL course on YouTube. He talks about environment internal state $S^e_t$, and agent internal state $S^a_t$. We know that state $s$ is Markov if $\mathbb{P}\{S_t=s|S_{...
3
votes
0answers
41 views

Why do we discount the state distribution?

In Reinforcement Learning, it is common to use a discount factor $\gamma$ to give less importance to future rewards when calculating the returns. I have also seen mention of discounted state ...
1
vote
0answers
17 views

Is there any research on the application of policy gradients to problems where the selection of an action requires the selection of another one?

I am working on a problem and want to explore if it can be solved with PPO (or other policy gradient methods). The problem is that the action space is a bit special, compared to classic RL ...
3
votes
1answer
53 views

What is the advantage of using MCTS with value based methods over value based methods only?

I have been trying to understand why MCTS is very important to the performance of RL agents, and the best description I found was from the paper Bootstrapping from Game Tree Search stating: ...
2
votes
0answers
38 views

Off-policy Bellman Operators: Writing Operator and Weight Update Function for a 2-State System

I am studying for RL on my own and was trying to solve this question I came across. Write an operator function $T(w, \pi, \mu, l, g)$ that takes weights $w$, a target policy $\pi$, a behaviour policy ...
0
votes
1answer
41 views

Given a sequence of states followed by the agent, is it possible to find the Q-value for a state-action pair not in this sequence?

Assume you are given a sequence of states followed by the agent, generated by a random policy, $[s_0, s_1, s_2, \dots, s_n]$. Furthermore, assume the MDP is fully observable and time is discrete. Is ...
1
vote
1answer
86 views

Why does the implementation of REINFORCE algorithm minimize the gradient term but not the loss?

I read the book "Foundation of Deep Reinforcement Learning, Laura Graesser and Wah Loon Keng", and when I go through the REINFORCE algorithm, they show the objective function: $$ J\left(\...
1
vote
0answers
14 views

Understanding advantage estimator in proximal policy optimization

I was reading Proximal Policy Optimization paper. It states following: The advantage estimator used is: $\hat{A}_t=-V(s_t)+r_t+\gamma r_{t+1}+...+\gamma^{T-t+1}r_{T-1}+\color{blue}{\gamma^{T-t}}V(s_T)...
6
votes
2answers
104 views

Are there RL algorithms that also try to predict the next state?

So far I've developed simple RL algorithms, like Deep Q-Learning and Double Deep Q-Learning. Also, I read a bit about A3C and policy gradient but superficially. If I remember correctly, all these ...
0
votes
0answers
13 views

Why has PILCO not been included in Sutton & Barto?

PILCO is a model-based Reinforcement Learning method introduced in 2011 by Deisenroth and Rasmussen. As far as I know, it is still considered one of the most important RL method, especially for its ...

1
2
3 4 5
35