Questions tagged [reinforcement-learning]

For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.

572 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
1
vote
0answers
30 views

Should we multiply the target of actor by the importance sampling ratio when prioritized replay is applied to DDPG?

According to PER, we have to multiply the $Q$ error $\delta_i$ by the importance sampling ratio to correct the bias introduced by the imbalance sampling of PER, where importance sampling ratio is ...
1
vote
0answers
71 views

How do I apply batch normalization to my algorithm?

I am working lately on batch normalization for the last couple of days, but I just can't seem to solve it on how to apply it. So, to give the full code to you, I will just send the implementation. ...
1
vote
0answers
216 views

How the actor use the output from the critic to make action in actor-critic network?

I am reading about the actor-critic architecture. I am confused about how the actor determines the action using the value (or future reward) from the critic network. Below you have the most popular ...
1
vote
0answers
66 views

How to make episode ending “good” in reinforcement learning?

TL;DR: read the bold. The rest are details I am trying to implement Reinforcement Learning:An Introduction, section 13.5 myself: on OpenAi's cartpole The algorithm seems to be learning something ...
1
vote
0answers
21 views

How to train chat bot on infinite non-stationary data?

I have continual simulated data of million sentences of two simulated persons talking to each other in a room and I want to model one of the persons speech. Now, during this period things in the room ...
1
vote
0answers
40 views

Where does the expectation term in the derivative of the soft-max policy come from?

At slide 17 of the David Silver's series, the soft-max policy is defined as follows $$ \pi_\theta(s, a) \propto e^{\phi(s, a)^T \theta} $$ that is, the probability of an action $a$ (in state $s$) is ...
1
vote
0answers
50 views

What limitations does the Markov property place on real time learning?

The Markov property is the dependence of a system's future state probability distribution solely on the present state, excluding any dependence on past system history. The presence of the Markov ...
1
vote
0answers
137 views

Reward discounting in reinforcement learning for a Pong game

I am trying to understand how to train a neural network to win a Pong game using reinforcement learning, by following the blog post Spinning up a Pong AI with deep reinforcement learning. The ...
1
vote
0answers
402 views

Continuous Advantage Actor Critic Implementation

I'm having trouble implementing AC for continuous action space. As far as I can tell, my code doesn't seem to have any bugs! The agent is learning "something" as its behaviour seems to vary ...
1
vote
0answers
697 views

Is my understanding of the differences between MDP, Semi MDP and POMDP correct?

I just wanted to confirm that my understanding of the different Markov Decision Processes are correct, because they are the fundamentals of reinforcement learning. Also, I read a few literature ...
1
vote
0answers
47 views

How to design the reward for an action which is the only legal action at some state

I am working on a RL project,but got stuck at one point: The task is continuous (Non-episodic). Following some suggestion from Sutton's RL book, I am using a value function approximation method with ...
1
vote
0answers
62 views

Reinforcement learning for segmenting the robot path to reflect the true distances

I have a grid of rectangles acting as blocks. The robot traverses through the inter-spaces between these consecutive blocks. Now I have sensor data streaming in representing Right and left wheel ...
1
vote
0answers
16 views

Does inflation should occur in output layer when I do Artificial Neural Network to increase smartness of the model?

The idea that come to my mind is called Value Based Model for ANN. We use simple DCF formula to calculate kind of Q value: Rewards/Discount rate. Discount rate is a risk of getting the reward on the ...
1
vote
0answers
49 views

Optimization step in Apprenticeship Learning via Inverse Reinforcement Learning

Why the optimization step of the algorithm a quadratic program? [See: Apprenticeship Learning via Inverse Reinforcement Learning; page 3] Isn't the objective function linear? Why don't we treat ...
1
vote
0answers
16 views

Can you make the first layer of a net have discernible shapes?

Coming from the YT videos of 3blue1brown which showed that the individual layers do not have discernible shapes in the case of hand written letter recognition, I wondered if you could penalize ...
1
vote
0answers
68 views

Learning from events. Supervised, Unsupervised or MDP?

I have a large set of simulation logs for a market simulation of which I want to learn from. The market includes: customers products (subscriptions) The customers choose products and then stick with ...
1
vote
0answers
30 views

Name of a multiarmed bandit with only some levers available

In order to model a card game, as an exercise, I was thinking of an elementary setting as a multiarmed bandit, each lever being the distribution of expected rewards of a specific card. But, of course,...
1
vote
0answers
38 views

Dealing with input to recurrent net with changing dimensions

I have a problem in which the dimensions of the input are increasing in row and column at each timestep. What method for preprocessing could be done or are there any architectures used for solving ...
1
vote
0answers
106 views

When do you back-propagate errors through a neural network when using TD($\lambda$)?

I have a neural network that I'm want to use to self-play Connect Four. The neural network receives the board state and is to provide an estimate of the state's value. I would then, for each move, ...
1
vote
0answers
66 views

Handling varied-size input with fixed-input network

I'm running A3C (Asynchronous Actor-Critic Agents) to learn a game where an agent needs to catch 3 rewards. The input of my network, among other things, is the relative position of the 3 rewards ...
1
vote
0answers
88 views

Agent exploration which leads to a negative state where actions are limited

I'm working on a project where I train a Q-learning agent to learn an optimal control policy for a water heater. I've set up a simulation which allows the agent to explore for one year. I then examine ...
0
votes
0answers
6 views

Maintaing values table for complex state and action space

I need to write value function for a complex environment in Python. Each state is a large tuple and each action is a Numpy array. If they both are integers, then I can implement the value function as ...
0
votes
0answers
15 views

Reinforcement Learning: How to train policy model continuously the incremental way

I'm using a DNN model to keep the policy of a reinforcement learning agent, it maps a certain state to suitable action. However, after every action of the agent with good reward, the training data are ...
0
votes
0answers
10 views

What is the best way to convert an historical log data file to a Markov Decision Process (MDP) to perform Q-learning?

Hypothetically, I have an historical log file whose entries contain the instantaneous throughput for the transfer of a set of files (25,000 files ranging in size from 101KB to 222MB) recorded every ...
0
votes
1answer
38 views

What is the difference between a fitness function and a reward function?

In reinforcement learning (RL), the reward function (RF), which can be denoted as $r(s)$, $r(s, a)$, $r(s, a, s')$, $r(s, s')$ depending on its specific definition, provides the learning signal, which ...
0
votes
0answers
48 views

Is my reward function non-Markovian?

I am working on an RL problem where the time when the agent obtains the reward for taking action $a$ in time step $t$ is stochastic. In fact, there is no immediate reward for taking action $a$ in time ...
0
votes
0answers
17 views

Which loss function should I use to train DDGP with multiple q values, one for each of the output dimensions?

I'm trying to come up with a loss function for the case, in DDPG, where we have as many outputs from the critic as there are from the actor. So, there will be one Q value for each dimension in the ...
0
votes
0answers
34 views

How to evaluate the learning effect of reinforcement learning

I am a student who has just started learning about reinforcement learning. Is there a way to evaluate the learning effect of reinforcement learning that can be actually calculated from information ...
0
votes
0answers
17 views

Neuroevolution + RL: How to make sure my policies are more diverse?

I currently implemented Deep Neuroevolution and used it on a couple of Atari games. For my implementation I used a similar Genetic Algorithm, network and setup as the Uber AI Deep Neuroevolution paper ...
0
votes
0answers
40 views

What are good techniques for continuous learning in production?

I was wondering which AI techniques and architectures are used in environments that need predictions to continually improve by the feedback of the user. So let's take some kind of recommendation ...
0
votes
0answers
36 views

Is training on single game each time appropriate for an agent to learn to play checkers

I was facing a problem I mentioned in a previous question but after a while, I realize that maybe the problem in the dataset not in the learning rate. I build the dataset from white positions only i.e ...
0
votes
0answers
15 views

Is there any solution to the problem of detecting whether a user is having trouble finding something while surfing a webpage?

While a user is navigating through a website, we need to detect whether the user is having trouble finding something in realtime. The output is used to trigger an event that should pop up a FAQ page ...
0
votes
0answers
46 views

Is there any toy example that can exemplify the performance of double Q-learning?

I recently tried to reproduce the results of double Q-learning. However, the results are not satisfying. I have also tried to compare double Q learning with Q-learning in Taxi-v3, FrozenLake without ...
0
votes
0answers
32 views

In AlphaZero, which features are one-hot encoded and which are single real-valued?

From the AlphaZero paper, the caption of Table S1 (p. 13) Table S1: Input features used by AlphaZero in Go, Chess and Shogi respectively. The first set of features are repeated for each position in a ...
0
votes
0answers
38 views

How to implement RL model with increasing dimensions of state space and action space?

I've read in this discussion that "reinforcement learning is a way of finding the value function of a Markov Decision Process". I want to implement an RL model, whose state space and action ...
0
votes
0answers
15 views

Using an LSTM for model-based RL in a POMDP

I am trying to set up an experiment where an agent is exploring an n x n gridworld environment, of which the agent can see some fraction at any given time step. I'd like the agent to build up some ...
0
votes
0answers
14 views

Suppose every-visit MC was used instead of first-visit MC on blackjack. Would you expect the results to be different?

This is a question from page 94 of Sutton and Barto's RL book 2020. I read in someone's compiled GitHub answers to this book's exercises their answer was: "No because each state in an episode of ...
0
votes
0answers
21 views

Hierarchical reinforcement learning for combinatorial complexity

I want to try a hierarchical reinforcement learning (HRL) approach to hard logical problems with combinatorial complexity, i.e. games like chess or Rubik's cube. The majority of HRL papers I have ...
0
votes
0answers
33 views

DQN fails to learn useful policy for the Taxi environment (Dietterich 200)

I'm building an agent to solve the Taxi environment. I've seen this problem solved with Q-Learning algorithms but my DQN consistently fails to learn anything. The environment has a discrete ...
0
votes
0answers
26 views

Is there a resource that explains which settings mean 'High' or 'Low' difficulty in the ALE environment?

I have been using AIgym to train my RL agents. I am now trying to take advantage of the different difficulty settings that the ALE offers. However I can't find a resource that explains which ...
0
votes
0answers
34 views

Thompson sampling with Bernoulli prior and non-binary reward update

I am solving a problem for which I have to select the best possible servers (level 1) to hit for a given data. These servers (level 1) in turn hit some other servers (level 2) to complete the request. ...
0
votes
0answers
27 views

What are the implications of storing the alternative situation (that could have been experienced) in the replay buffer?

Consider an environment where there are 2 outcomes (e.g. dead and alive) and a discrete set of actions. For example, a game where the agent has 2 guns $A$ and $B$ to shoot a monster (the monster dies ...
0
votes
0answers
18 views

Need suggestion for Reinforcement Learning based visual landing system for quadcopters (UAVs)

I have deep interest in quadcopters. I need ideas for designing of an experiment. I have a programmable quadcopter. I can autonomously land it on a staitonary landing pad with a vision algorithm. ...
0
votes
0answers
43 views

Off-policy full-random training in easy-to-explore environment

Let say we are in an environment where a random agent can easily explore all the states of an environment (for example: tic-tac-toe). In those environments, using off-policy algorithm, is it a good ...
0
votes
1answer
43 views

How can reinforcement learning be applied when the goal location or environment is unknown?

I am studying RL. I was thinking whether a new state value or the observation is provided by the environment before the agent actually implements the action. Take the maze problem as an example. Each ...
0
votes
0answers
38 views

Is using Bellman Optimality Equation to evaluate states a bad idea when episode number is low?

I am trying to build an RL agent that interacts with an environment, a 2D grid of dimensions 20*10: each (i,j) square in the grid gives out some reward to the agent when it visits that square. Each ...
0
votes
0answers
20 views

Relative Weighting of Loss Weights for Self-Play Reinforcement Learning

I am training some self play reinforcement learning agents to play 2 player board games like Connect 4, Othello, and The Game of the Amazons. For each game, there is a single neural network with 2 ...
0
votes
0answers
40 views

Variance of the Gaussian policy is not decreasing while training the agent using Soft Actor-Critic method

I've written my own version of SAC(v2) for a problem with continuous action space. While training, the losses for the value network and both q functions steadily decrease down to 0.02-0.03. The loss ...
0
votes
0answers
23 views

Reinforcement comparison optimality

The following is definition of reinforcement comparison, which updates an average reward and a preference for each action http://incompleteideas.net/book/first/ebook/node22.html I want to know if this ...
0
votes
1answer
26 views

Why would the reward of A3C with LSTM suddenly drop off after many episodes?

I am training an A3C with stacked LSTM. During initial training, my model was giving descent +ve reward. However, after many episodes, its reward just goes to zero and is continuing for a long time. ...

1
8 9
10
11 12