Questions tagged [reinforcement-learning]
For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.
572
questions with no upvoted or accepted answers
1
vote
0answers
30 views
Should we multiply the target of actor by the importance sampling ratio when prioritized replay is applied to DDPG?
According to PER, we have to multiply the $Q$ error $\delta_i$ by the importance sampling ratio to correct the bias introduced by the imbalance sampling of PER, where importance sampling ratio is ...
1
vote
0answers
71 views
How do I apply batch normalization to my algorithm?
I am working lately on batch normalization for the last couple of days, but I just can't seem to solve it on how to apply it. So, to give the full code to you, I will just send the implementation.
...
1
vote
0answers
216 views
How the actor use the output from the critic to make action in actor-critic network?
I am reading about the actor-critic architecture. I am confused about how the actor determines the action using the value (or future reward) from the critic network.
Below you have the most popular ...
1
vote
0answers
66 views
How to make episode ending “good” in reinforcement learning?
TL;DR: read the bold. The rest are details
I am trying to implement Reinforcement Learning:An Introduction, section 13.5 myself:
on OpenAi's cartpole
The algorithm seems to be learning something ...
1
vote
0answers
21 views
How to train chat bot on infinite non-stationary data?
I have continual simulated data of million sentences of two simulated persons talking to each other in a room and I want to model one of the persons speech. Now, during this period things in the room ...
1
vote
0answers
40 views
Where does the expectation term in the derivative of the soft-max policy come from?
At slide 17 of the David Silver's series, the soft-max policy is defined as follows
$$
\pi_\theta(s, a) \propto e^{\phi(s, a)^T \theta}
$$
that is, the probability of an action $a$ (in state $s$) is ...
1
vote
0answers
50 views
What limitations does the Markov property place on real time learning?
The Markov property is the dependence of a system's future state probability distribution solely on the present state, excluding any dependence on past system history.
The presence of the Markov ...
1
vote
0answers
137 views
Reward discounting in reinforcement learning for a Pong game
I am trying to understand how to train a neural network to win a Pong game using reinforcement learning, by following the blog post
Spinning up a Pong AI with deep reinforcement learning.
The ...
1
vote
0answers
402 views
Continuous Advantage Actor Critic Implementation
I'm having trouble implementing AC for continuous action space. As far as I can tell, my code doesn't seem to have any bugs! The agent is learning "something" as its behaviour seems to vary ...
1
vote
0answers
697 views
Is my understanding of the differences between MDP, Semi MDP and POMDP correct?
I just wanted to confirm that my understanding of the different Markov Decision Processes are correct, because they are the fundamentals of reinforcement learning. Also, I read a few literature ...
1
vote
0answers
47 views
How to design the reward for an action which is the only legal action at some state
I am working on a RL project,but got stuck at one point: The task is continuous (Non-episodic). Following some suggestion from Sutton's RL book, I am using a value function approximation method with ...
1
vote
0answers
62 views
Reinforcement learning for segmenting the robot path to reflect the true distances
I have a grid of rectangles acting as blocks. The robot traverses through the inter-spaces between these consecutive blocks. Now I have sensor data streaming in representing Right and left wheel ...
1
vote
0answers
16 views
Does inflation should occur in output layer when I do Artificial Neural Network to increase smartness of the model?
The idea that come to my mind is called Value Based Model for ANN. We use simple DCF formula to calculate kind of Q value: Rewards/Discount rate. Discount rate is a risk of getting the reward on the ...
1
vote
0answers
49 views
Optimization step in Apprenticeship Learning via Inverse Reinforcement Learning
Why the optimization step of the algorithm a quadratic program? [See: Apprenticeship Learning via Inverse Reinforcement Learning; page 3]
Isn't the objective function linear? Why don't we treat ...
1
vote
0answers
16 views
Can you make the first layer of a net have discernible shapes?
Coming from the YT videos of 3blue1brown which showed that the individual layers do not have discernible shapes in the case of hand written letter recognition, I wondered if you could penalize ...
1
vote
0answers
68 views
Learning from events. Supervised, Unsupervised or MDP?
I have a large set of simulation logs for a market simulation of which I want to learn from. The market includes:
customers
products (subscriptions)
The customers choose products and then stick with ...
1
vote
0answers
30 views
Name of a multiarmed bandit with only some levers available
In order to model a card game, as an exercise, I was thinking of an elementary setting as a multiarmed bandit, each lever being the distribution of expected rewards of a specific card.
But, of course,...
1
vote
0answers
38 views
Dealing with input to recurrent net with changing dimensions
I have a problem in which the dimensions of the input are increasing in row and column at each timestep. What method for preprocessing could be done or are there any architectures used for solving ...
1
vote
0answers
106 views
When do you back-propagate errors through a neural network when using TD($\lambda$)?
I have a neural network that I'm want to use to self-play Connect Four. The neural network receives the board state and is to provide an estimate of the state's value.
I would then, for each move, ...
1
vote
0answers
66 views
Handling varied-size input with fixed-input network
I'm running A3C (Asynchronous Actor-Critic Agents) to learn a game where an agent needs to catch 3 rewards. The input of my network, among other things, is the relative position of the 3 rewards ...
1
vote
0answers
88 views
Agent exploration which leads to a negative state where actions are limited
I'm working on a project where I train a Q-learning agent to learn an optimal control policy for a water heater. I've set up a simulation which allows the agent to explore for one year. I then examine ...
0
votes
0answers
6 views
Maintaing values table for complex state and action space
I need to write value function for a complex environment in Python.
Each state is a large tuple and each action is a Numpy array.
If they both are integers, then I can implement the value function as ...
0
votes
0answers
15 views
Reinforcement Learning: How to train policy model continuously the incremental way
I'm using a DNN model to keep the policy of a reinforcement learning agent, it maps a certain state to suitable action.
However, after every action of the agent with good reward, the training data are ...
0
votes
0answers
10 views
What is the best way to convert an historical log data file to a Markov Decision Process (MDP) to perform Q-learning?
Hypothetically, I have an historical log file whose entries contain the instantaneous throughput for the transfer of a set of files (25,000 files ranging in size from 101KB to 222MB) recorded every ...
0
votes
1answer
38 views
What is the difference between a fitness function and a reward function?
In reinforcement learning (RL), the reward function (RF), which can be denoted as $r(s)$, $r(s, a)$, $r(s, a, s')$, $r(s, s')$ depending on its specific definition, provides the learning signal, which ...
0
votes
0answers
48 views
Is my reward function non-Markovian?
I am working on an RL problem where the time when the agent obtains the reward for taking action $a$ in time step $t$ is stochastic. In fact, there is no immediate reward for taking action $a$ in time ...
0
votes
0answers
17 views
Which loss function should I use to train DDGP with multiple q values, one for each of the output dimensions?
I'm trying to come up with a loss function for the case, in DDPG, where we have as many outputs from the critic as there are from the actor. So, there will be one Q value for each dimension in the ...
0
votes
0answers
34 views
How to evaluate the learning effect of reinforcement learning
I am a student who has just started learning about reinforcement learning.
Is there a way to evaluate the learning effect of reinforcement learning that can be actually calculated from information ...
0
votes
0answers
17 views
Neuroevolution + RL: How to make sure my policies are more diverse?
I currently implemented Deep Neuroevolution and used it on a couple of Atari games. For my implementation I used a similar Genetic Algorithm, network and setup as the Uber AI Deep Neuroevolution paper ...
0
votes
0answers
40 views
What are good techniques for continuous learning in production?
I was wondering which AI techniques and architectures are used in environments that need predictions to continually improve by the feedback of the user.
So let's take some kind of recommendation ...
0
votes
0answers
36 views
Is training on single game each time appropriate for an agent to learn to play checkers
I was facing a problem I mentioned in a previous question but after a while, I realize that maybe the problem in the dataset not in the learning rate.
I build the dataset from white positions only i.e ...
0
votes
0answers
15 views
Is there any solution to the problem of detecting whether a user is having trouble finding something while surfing a webpage?
While a user is navigating through a website, we need to detect whether the user is having trouble finding something in realtime. The output is used to trigger an event that should pop up a FAQ page ...
0
votes
0answers
46 views
Is there any toy example that can exemplify the performance of double Q-learning?
I recently tried to reproduce the results of double Q-learning. However, the results are not satisfying. I have also tried to compare double Q learning with Q-learning in Taxi-v3, FrozenLake without ...
0
votes
0answers
32 views
In AlphaZero, which features are one-hot encoded and which are single real-valued?
From the AlphaZero paper, the caption of Table S1 (p. 13)
Table S1: Input features used by AlphaZero in Go, Chess and Shogi respectively. The first set of features are repeated for each position in a ...
0
votes
0answers
38 views
How to implement RL model with increasing dimensions of state space and action space?
I've read in this discussion that "reinforcement learning is a way of finding the value function of a Markov Decision Process".
I want to implement an RL model, whose state space and action ...
0
votes
0answers
15 views
Using an LSTM for model-based RL in a POMDP
I am trying to set up an experiment where an agent is exploring an n x n gridworld environment, of which the agent can see some fraction at any given time step. I'd like the agent to build up some ...
0
votes
0answers
14 views
Suppose every-visit MC was used instead of first-visit MC on blackjack. Would you expect the results to be different?
This is a question from page 94 of Sutton and Barto's RL book 2020.
I read in someone's compiled GitHub answers to this book's exercises their answer was: "No because each state in an episode of ...
0
votes
0answers
21 views
Hierarchical reinforcement learning for combinatorial complexity
I want to try a hierarchical reinforcement learning (HRL) approach to hard logical problems with combinatorial complexity, i.e. games like chess or Rubik's cube. The majority of HRL papers I have ...
0
votes
0answers
33 views
DQN fails to learn useful policy for the Taxi environment (Dietterich 200)
I'm building an agent to solve the Taxi environment. I've seen this problem solved with Q-Learning algorithms but my DQN consistently fails to learn anything. The environment has a discrete ...
0
votes
0answers
26 views
Is there a resource that explains which settings mean 'High' or 'Low' difficulty in the ALE environment?
I have been using AIgym to train my RL agents. I am now trying to take advantage of the different difficulty settings that the ALE offers.
However I can't find a resource that explains which ...
0
votes
0answers
34 views
Thompson sampling with Bernoulli prior and non-binary reward update
I am solving a problem for which I have to select the best possible servers (level 1) to hit for a given data. These servers (level 1) in turn hit some other servers (level 2) to complete the request. ...
0
votes
0answers
27 views
What are the implications of storing the alternative situation (that could have been experienced) in the replay buffer?
Consider an environment where there are 2 outcomes (e.g. dead and alive) and a discrete set of actions. For example, a game where the agent has 2 guns $A$ and $B$ to shoot a monster (the monster dies ...
0
votes
0answers
18 views
Need suggestion for Reinforcement Learning based visual landing system for quadcopters (UAVs)
I have deep interest in quadcopters. I need ideas for designing of an experiment. I have a programmable quadcopter. I can autonomously land it on a staitonary landing pad with a vision algorithm. ...
0
votes
0answers
43 views
Off-policy full-random training in easy-to-explore environment
Let say we are in an environment where a random agent can easily explore all the states of an environment (for example: tic-tac-toe).
In those environments, using off-policy algorithm, is it a good ...
0
votes
1answer
43 views
How can reinforcement learning be applied when the goal location or environment is unknown?
I am studying RL. I was thinking whether a new state value or the observation is provided by the environment before the agent actually implements the action.
Take the maze problem as an example. Each ...
0
votes
0answers
38 views
Is using Bellman Optimality Equation to evaluate states a bad idea when episode number is low?
I am trying to build an RL agent that interacts with an environment, a 2D grid of dimensions 20*10: each (i,j) square in the grid gives out some reward to the agent when it visits that square. Each ...
0
votes
0answers
20 views
Relative Weighting of Loss Weights for Self-Play Reinforcement Learning
I am training some self play reinforcement learning agents to play 2 player board games like Connect 4, Othello, and The Game of the Amazons.
For each game, there is a single neural network with 2 ...
0
votes
0answers
40 views
Variance of the Gaussian policy is not decreasing while training the agent using Soft Actor-Critic method
I've written my own version of SAC(v2) for a problem with continuous action space. While training, the losses for the value network and both q functions steadily decrease down to 0.02-0.03. The loss ...
0
votes
0answers
23 views
Reinforcement comparison optimality
The following is definition of reinforcement comparison, which updates an average reward and a preference for each action
http://incompleteideas.net/book/first/ebook/node22.html
I want to know if this ...
0
votes
1answer
26 views
Why would the reward of A3C with LSTM suddenly drop off after many episodes?
I am training an A3C with stacked LSTM.
During initial training, my model was giving descent +ve reward. However, after many episodes, its reward just goes to zero and is continuing for a long time. ...
