Questions tagged [reinforcement-learning]
For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.
1,697
questions
0
votes
0answers
18 views
Using neural network approximator in reinforcementlearning.jl [closed]
I am trying to create a simultaneous multi agent environment using reinforcementlearning.jl
I have successfully represented the environment and it works with a RandomPolicy for every agent.
But my ...
0
votes
0answers
20 views
State-Space Representation for A Knapsack-like Optimization Problem
I am trying to implement a solution to my thesis problem statement -a task scheduling optimization- by using DQN and Deep SARSA. It is essentially very similar to the 0-1 knapsack problem: there are ...
2
votes
1answer
37 views
Is existence and uniqueness of state-value function at $\gamma < 1$ theoretical?
Consider the following statement from 4.1 Policy Evaluation of the first edition of Sutton and Barto's book.
The existence and uniqueness of $V^{\pi}$ are guaranteed as long as
either $\gamma < 1$...
1
vote
0answers
14 views
Is there any thumb rule on the cardinality of state space in order to use the parameterized function to estimate value functions?
Value functions for a given MDP can be learned in at least two ways by experience.
The first way (tabular calculation) is generally used in the case of state spaces that are small enough.
The second ...
-1
votes
0answers
38 views
Chatbot for mental health [closed]
I want to build a chatbot to help people who are depressed, anxious, or chronically sad. My idea is to build a chatbot with Deep RL that behaves like a therapist and helps the person in a simple ...
3
votes
1answer
147 views
How is the state-value function expressed as a product of sums?
The state-value function for a given policy $\pi$ is given by
$$\begin{align}
V^{\pi}(s) &=E_{\pi}\left\{r_{t+1}+\gamma r_{t+2}+\gamma^{2} r_{t+3}+\cdots \mid s_{t}=s\right\} \\
&=E_{\pi}\...
4
votes
1answer
58 views
What are the major differences between multi-armed bandits and the other well-known algorithms (DQN, A3C, PPO, etc)?
I have studied in the past different algorithms, i.e. DQN, DDQN, REINFORCE, A3C, PPO, TRPO, so on. I am doing an internship this summer where I have to use a multi-armed bandit (MAB). I am a bit ...
0
votes
0answers
20 views
Reward firstly increase, but after more episodes, start decrease, and weights diverges
I'm making a simple deep Q learning algorithm, with cartpole-v1 env.
Like you can see in chart, after many episodes the reward decrease, some possible reasons?
The exploration vs axplotation algorithm ...
0
votes
0answers
9 views
DQN + HER, TD Error spiked then success rate plummets. What went wrong?
TL;DR:
I trained a DQN + HER model using stable-baselines library for a custom environment. I noticed that in most runs, sometimes the TD-Error will spike and then the success rate of my model ...
0
votes
0answers
12 views
Comparison between TD(0) and MC ( or GAE )?
I'm getting started with DRL and have trouble distinguishing TD(0), MC, and GAE; and which scenarios one's better than others. Here is what I understand so far:
TD(0): increment learning, can learn ...
0
votes
1answer
27 views
What would happen to an agent trained using Markov Decision Process if the goal node changes?
I was reading up a paper that did routing based on an MDP, and I was wondering because, in routing, there is a sender node and a receiver node, so if the receiver node changes (sending a message to ...
3
votes
0answers
75 views
How to deal with a moving target in the Lunar Lander environment with DDPG?
I have noticed that DDPG does rather well at solving environments with a static target.
For example, the default of Lunar Lander, the flags do not change position. So the DDPG model learns how to get ...
-1
votes
1answer
20 views
Is it ture to use cumulative reward instead of step reward? [closed]
The straightforward way of assigning reward for each step always has been the step's itself reward. But what if my objective is to have the highest episodic reward and try to assign cumulative reward ...
-1
votes
1answer
29 views
Exploration for softmax should be binary or continuous softmax?
Maybe it's silly to ask but for random exploration in an RL for choosing discrete action, that in the neural network last layer softmax will be used, what random samples should we provide? binary like ...
1
vote
1answer
29 views
PPO agent for vehicle control does not learn to stop at traffic lights
I have built a custom RL environment with gym, which simulates the RL vehicle and potential vehicles in front of the RL vehicle as well as traffic lights (including ...
-1
votes
0answers
29 views
finding the policy of a certain state (reinforcement learning)
Here is a gridworld. The black state is a wall/inaccessible. The red state is a state with -1 reward, if you reach this state, the episode ends. The aim is to go to the green state and the episode ...
2
votes
0answers
32 views
suitable reference for reinforcement learning for beginners [duplicate]
I am looking for a reference book on RL for first time learners, one that is a gentle introduction and not as wordy as Sutton/Barto. I am interested in something similar to Georgia Institute of ...
0
votes
0answers
46 views
Reinforcement learning for rearranging the mobile home screen icon layout: what inputs/states do I need to pass into the algorithm?
I have a problem where I need to rearrange a particular user's mobile home screen icon layout. Let's say that the social media app usage of a user is high compared to other app usage. So I need the ...
1
vote
0answers
38 views
Why isn't RL considered a continual learning strategy itself?
I have read about methods that apply continual learning strategies to reinforcement learning.
Since reinforcement learning also learns step by step (i.e., task by task, in a sense) during the training ...
2
votes
0answers
29 views
Is there any reasonable notion of regret for infinite horizon discounted MDPs?
I am thinking about episodic MDPs. Usually, in episodic MDPs, it seems that we have a finite fixed horizon per episode and no discount factor. Then, a very intuitive notion of regret after $T$ ...
3
votes
1answer
77 views
In DQN, would it be cheaper to have $N$ neural networks with a single real-valued output, one for each of the $N$ actions?
In the classical examples of deep q-learning, I often see neural networks in which the input represents the state of the agent, while the output is a tuple with all the values of $Q(s, a)$ predicted ...
0
votes
0answers
30 views
Should the actor and critic share a common feature extraction neural network?
In an environment with image observations, if we use an actor-critic method to find a good policy, commonly, we will use a feature extraction neural network, such as ResNet, to extract the information ...
1
vote
1answer
42 views
How to scale all positive continuous reward?
My RL project has all positive continuous rewards for every step and the goal is to have the maximum cumulative reward (episodic reward). The problem is that the rewards are too close and all between ...
1
vote
0answers
36 views
PPO in continuous control not working
I have PPO agent for discrete action space for LunarLander-v2 env in gym and it works well. However, when i am trying to solve continuous version of the same env - <...
0
votes
0answers
17 views
What are the state-of-the-art learning algorithms for contextual bandits with stochastic rewards
I am building a solution for an environment with stochastic rewards in an online setting. I am wondering what the state of the art is in this setting.
Is it $\epsilon$-greedy (with logistic regression)...
0
votes
1answer
16 views
Why is it that the state visitation frequency equals the sum of state visitation frequency from initial time step to the horizon?
In the maximum entropy inverse reinforcement learning paper, Ziebart et al. show that the state visitation frequency $\rho(s)$ of a state $s$ can be computed as
$$
\rho_{\pi}(s) = \sum_{t}^{T} P(s_t=s|...
2
votes
1answer
60 views
Difference in UCB performance when scaling the rewards
I notice the following behavior when running experiments with $\epsilon$-greedy and UCB1. If the reward is kept binary (0 or 1) both algorithm's performances are on par with each other. However, if I ...
0
votes
0answers
6 views
Update state representation based on agent action
I am working on a project where I have to train a RL agent which will simulate Loan repayment track of a customer's loan based on his features derived from his credit profile (state vector). I am ...
2
votes
2answers
87 views
Why does Alpha Zero's Neural Network flip the board to be oriented towards the current player?
While reading the AlphaZero paper in preparation to code my own RL algorithm to play Chess decently well, I saw that the
"The board is oriented to the perspective of the current player."
I ...
1
vote
1answer
45 views
What would be the importance sampling ratio for off-policy TD learning control using Q values?
The off-policy TD learning control using state value function from page 34 of David Silver's RL lecture is:
$$ V(S_t) \leftarrow V(S_t) + \alpha \left( \frac{ \pi(A_t|S_t)}{\mu (A_t|S_t)} (R_{t+1} + \...
-2
votes
0answers
26 views
RL with tabular state instead of image
Looking through PyTorch's examples on RL and noticed that the state of the environment is always inferred by a CNN on images. Is it possible to do the same procedure where the state is instead a table ...
-1
votes
1answer
25 views
Using states (features) and actions from a heuristic model to estimate the value function of a reinforcement learning agent [closed]
new to RL here.
As far as i understood from RL courses, that there is two sides of reinforcement learning. Policy Evaluation, which is the task of knowing the value function for certain policy. and ...
1
vote
0answers
31 views
Is this a supervised or reinforcement learning problem, and which algorithm should I use to solve it?
I have a time series data with a little unusual cost/reward function (I haven't seen it before)
The model must predict a $Y$ value for any $X(t)$.
The reward is computed as follows. The model will ...
-1
votes
0answers
39 views
Can I create a Q-table in iterated procedure?
Let's say I have a time series vector (hourly price data), and want to obtain an optimal trading policy. To do so, I need a Q-table. My question is about creating such table from a raw time-series.
...
3
votes
1answer
75 views
How does the Alpha Zero's move encoding work?
I am a beginner in AI. I'm trying to train a multi-agent RL algorithm to play chess. One issue that I ran into was representing the action space (legal moves/or honestly just moves in general) ...
0
votes
0answers
48 views
Open AI Taxi - Agent fails to learn an effective policy
I'm trying to solve the openai gym taxi problem (v3) using deep q learning. I've already had some success with the q-table approach, but for the life of me cannot manage to train a NN to learn a ...
0
votes
0answers
29 views
Any RL approaches for this 2D space optimisation problem?
I have a list of rectangles, they are in certain order in 2D at the beginning. The task is to move them to get the boundary (rectangular) of the minimal area. It's OK to push off the dotted border as ...
5
votes
2answers
179 views
Does the policy iteration convergence hold for finite-horizon MDP?
Most RL books (Sutton & Barto, Bertsekas, etc.) talk about policy iteration for infinite-horizon MDPs. Does the policy iteration convergence hold for finite-horizon MDP? If yes, how can we derive ...
0
votes
0answers
20 views
Can I use binary numbers and plain floats in the same neural network input for reinforcement learning?
I'm modeling a system whose configuration can be represented by a binary array ([1,0,0,1] or [0,1,0,0], for example), and the agent can move on a 2D space (thus having 3 DOF), and the action the agent ...
1
vote
0answers
57 views
Relation between discounted MDP and stochastic shortest path problems in RL
I have been reading about discounted MDPs and Stochastic Shortest Path (SSP). I recently came to know (from a friend) that every discounted MDP can be converted to an equivalent SSP but not the other ...
1
vote
1answer
37 views
How to properly resume training of deep Q-learning network?
I'm currently training a deep q-learning network. Due to resource limitations, I am not able to train the model to the desired performance in one go. So what I'm doing now is training the model for a ...
0
votes
0answers
27 views
Double DQN backpropagation of negative final rewards?
My problem is that in my Double DQN model, negative final rewards are not being backpropagated into action Q-values, and so some Q-values are positive, when they should be negative, and hence ...
2
votes
2answers
146 views
With Monte Carlo off-policy learning what do we correct by using importance sampling?
I do not understand the link of importance sampling to Monte Carlo off-policy learning.
We estimate a value using sampling on whole episodes, and we take these values to construct the target policy.
...
1
vote
1answer
32 views
Why is the logarithm of the standard deviation used in this implementation of proximal policy optimization?
I am currently writing my bachelor thesis, which is an implementation of proximal policy optimization. Sometimes, I hit a wall because of the gaps in my mathematical knowledge. However, implementing ...
2
votes
2answers
231 views
How to fight with unstability in self play?
I'm working on a neural network that plays some board games like reversi or tic-tac-toe (zero-sum games, two players). I'm trying to have one network topology for all the games - I specifically don't ...
2
votes
1answer
53 views
How to recover the target Q network's weights solely from the snapshots of the primary Q network's weights in DQN?
Suppose that I have a DQN agent, which has two neural networks: one is the primary Q network and the other is the target Q network. In every update, the target Q network is updated with a soft update ...
0
votes
0answers
8 views
Is it possible to use agent coordinates directly as part of the state
I am working on graph optimisation problem using DQN - the graph is represented as an adjacency matrix and an agent moves through this matrix removing edges between nodes (add a 0) or adding edges ...
0
votes
0answers
27 views
PyTorch: How to deal with hidden states of an LSTM?
I have a time series in which each date is correlated with the preview one, and base on that I am trying to predict action 1 and action 2. But the problem is that I am not sure how to deal with the <...
-2
votes
1answer
45 views
Why the optimal Bellman operator of a Q-function can be approximated by a single point
I am currently studying reinforcement learning, especially DQN.
In DQN, learning proceeds in such a way as to minimize the norm (least-squares, Huber, etc.) of the optimal Bellman equation and the ...
0
votes
0answers
29 views
Backpropagation in REINFORCE algorithms with Categorical / Multinomial Distribution
From a paper by Williams, I know in general how to backpropagate log-probabilities of chosen actions when applying the REINFORCE weight update rule.
However, I was wondering about a case not being ...
