Questions tagged [reinforcement-learning]

For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.

606 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
0
votes
0answers
15 views

How to have a DNN output a classification for each user at once?

I have a Reinforcement Learning environment with an agent that allocates power values to different users. To do so, I have thought of implementing a deep neural network like the one shown in the ...
0
votes
0answers
22 views

How is the discounted maximum entropy objective obtained for soft-q-learning and SAC

In the soft q-learning paper, they provide an expression for the maximum entropy objective that takes discounting into account. My main question is: can someone explain how they incorporated ...
0
votes
0answers
28 views

How to scale the action in a custom environment with DDPG?

I am trying to implement DDPG in a custom gym environment. The action is the relative allocation of funds between each asset. The action space is a Box with the ...
0
votes
0answers
25 views

Reinforcement learning parameterized action space

I've been working a RL problem with a parameterized action space: there is a finite set of discrete actions $A = \{a_1, a_2, ..., a_k\}$, and each $a_i$ is associated with a set of continuous-valued ...
0
votes
0answers
8 views

Do Learning Classifier Systems extend beyond reinforcement learning?

In 2000 the John Holland wrote concerning Learning Classifier Systems (LCS) In recent years there has been a focus on classifier systems as performance systems or evolutionary incarnations of ...
0
votes
0answers
7 views

What is an appropriate stop criteria for training on a non-stationary environment in reinforcement learning?

I'm currently studying reinforcement learning (RL) and would like to understand non-stationary environments better. So for stationary environments, the Q-values of all state-action pairs converge ...
0
votes
0answers
7 views

What is the rationale behind the minimap of MAgent?

The MAgent family of PettingZoo is based on a previous implementation that gives a little tutorial explaining the gridworld ...
0
votes
0answers
30 views

Proving existence or non existence of reward function to make given policy “uniquely” optimal when reward function is dependent only on S or both S,A

I was going through paper titled "Algorithms for Inverse Reinforcement Learning" by Andrew Ng and Russell. It states following basics: MDP $M$ is a tuple $(S,A,\{P_{sa}\},\gamma,R)$, where ...
0
votes
0answers
26 views

Control Strategy for Multi-Agent Systems

I am working on a smart grid system which can be modeled with multiple agents interacting with each other. The agents are physically coupled with non-linearities, the action of one agent has a direct ...
0
votes
0answers
22 views

How DynaQ behaves in stochastic world in comparison with other reinforcement learning algorithms?

I came across of implementations of a bunch of algorithms on stochastic windy gridworld. This is the graph comparing their performance: So clearly, it seems that DynaQ performs better than all other ...
0
votes
0answers
27 views

Understanding policies in helicopter control in the paper by Andrew Ng et al

I was going through this paper on helicopter flight control using reinforcement learning by Andrew Ng et al. It defines two policy classes to learn two policies, one for hovering the helicopter and ...
0
votes
0answers
36 views

How should I model the state and action spaces for a problem where the goal is to draw a line between two points?

I have a problem where the goal is for the agent to draw a single line between two points on a $500 \times 500$ white image. I have built my DQN. For now, the output layer's size of the network is $[...
0
votes
0answers
23 views

PPO: sampling next action vs picking the most probable action

According to the original Proximal Policy Optimization paper (PPO paper), we always sample an action from the actor distribution. According to the link The overall loss is calculated as $\text{loss} =...
0
votes
0answers
52 views

Is my reward function non-Markovian?

I am working on an RL problem where the time when the agent obtains the reward for taking action $a$ in time step $t$ is stochastic. In fact, there is no immediate reward for taking action $a$ in time ...
0
votes
0answers
17 views

Which loss function should I use to train DDGP with multiple q values, one for each of the output dimensions?

I'm trying to come up with a loss function for the case, in DDPG, where we have as many outputs from the critic as there are from the actor. So, there will be one Q value for each dimension in the ...
0
votes
0answers
17 views

Neuroevolution + RL: How to make sure my policies are more diverse?

I currently implemented Deep Neuroevolution and used it on a couple of Atari games. For my implementation I used a similar Genetic Algorithm, network and setup as the Uber AI Deep Neuroevolution paper ...
0
votes
0answers
45 views

What are good techniques for continuous learning in production?

I was wondering which AI techniques and architectures are used in environments that need predictions to continually improve by the feedback of the user. So let's take some kind of recommendation ...
0
votes
0answers
15 views

Is there any solution to the problem of detecting whether a user is having trouble finding something while surfing a webpage?

While a user is navigating through a website, we need to detect whether the user is having trouble finding something in realtime. The output is used to trigger an event that should pop up a FAQ page ...
0
votes
0answers
55 views

Is there any toy example that can exemplify the performance of double Q-learning?

I recently tried to reproduce the results of double Q-learning. However, the results are not satisfying. I have also tried to compare double Q learning with Q-learning in Taxi-v3, FrozenLake without ...
0
votes
0answers
53 views

In AlphaZero, which features are one-hot encoded and which are single real-valued?

From the AlphaZero paper, the caption of Table S1 (p. 13) Table S1: Input features used by AlphaZero in Go, Chess and Shogi respectively. The first set of features are repeated for each position in a ...
0
votes
0answers
65 views

Is my pseudocode titled “Monte Carlo Exploring Starts (with model)” correct?

Reinforcement Learning: An Introduction second edition, Richard S. Sutton and Andrew G. Barto: We made two unlikely assumptions above in order to easily obtain this guarantee of convergence for the ...
0
votes
0answers
67 views

How to frame this problem using RL?

How should this problem be framed in the domain of RL for preventing users from exceeding their bank account balance and being overdrawn? For example, a user has 1000 in an account, and proceeds to ...
0
votes
0answers
43 views

How to implement RL model with increasing dimensions of state space and action space?

I've read in this discussion that "reinforcement learning is a way of finding the value function of a Markov Decision Process". I want to implement an RL model, whose state space and action ...
0
votes
0answers
19 views

Using an LSTM for model-based RL in a POMDP

I am trying to set up an experiment where an agent is exploring an n x n gridworld environment, of which the agent can see some fraction at any given time step. I'd like the agent to build up some ...
0
votes
0answers
37 views

CNN for a DQN agent with a 2D matrix state and action as a 2D matrix

I have a custom environment, where the state is a 2D matrix of 11 rows (equals to number of users to satisfy) and 3 columns. Each column can take the value of either 0 or 1, and in each row, there can ...
0
votes
0answers
24 views

Suppose every-visit MC was used instead of first-visit MC on blackjack. Would you expect the results to be different?

This is a question from page 94 of Sutton and Barto's RL book 2020. I read in someone's compiled GitHub answers to this book's exercises their answer was: "No because each state in an episode of ...
0
votes
0answers
126 views

How should I build this DQN agent?

I have a set of users that can be one of 3 types. They will randomly request a service from the UAV which is a drone used as a Base Station. The UAV (the agent) is tasked with allocating subchannels (...
0
votes
0answers
24 views

Hierarchical reinforcement learning for combinatorial complexity

I want to try a hierarchical reinforcement learning (HRL) approach to hard logical problems with combinatorial complexity, i.e. games like chess or Rubik's cube. The majority of HRL papers I have ...
0
votes
0answers
34 views

DQN fails to learn useful policy for the Taxi environment (Dietterich 200)

I'm building an agent to solve the Taxi environment. I've seen this problem solved with Q-Learning algorithms but my DQN consistently fails to learn anything. The environment has a discrete ...
0
votes
0answers
26 views

Is there a resource that explains which settings mean 'High' or 'Low' difficulty in the ALE environment?

I have been using AIgym to train my RL agents. I am now trying to take advantage of the different difficulty settings that the ALE offers. However I can't find a resource that explains which ...
0
votes
0answers
36 views

Thompson sampling with Bernoulli prior and non-binary reward update

I am solving a problem for which I have to select the best possible servers (level 1) to hit for a given data. These servers (level 1) in turn hit some other servers (level 2) to complete the request. ...
0
votes
0answers
28 views

What are the implications of storing the alternative situation (that could have been experienced) in the replay buffer?

Consider an environment where there are 2 outcomes (e.g. dead and alive) and a discrete set of actions. For example, a game where the agent has 2 guns $A$ and $B$ to shoot a monster (the monster dies ...
0
votes
0answers
43 views

Off-policy full-random training in easy-to-explore environment

Let say we are in an environment where a random agent can easily explore all the states of an environment (for example: tic-tac-toe). In those environments, using off-policy algorithm, is it a good ...
0
votes
1answer
52 views

How can reinforcement learning be applied when the goal location or environment is unknown?

I am studying RL. I was thinking whether a new state value or the observation is provided by the environment before the agent actually implements the action. Take the maze problem as an example. Each ...
0
votes
0answers
41 views

Is using Bellman Optimality Equation to evaluate states a bad idea when episode number is low?

I am trying to build an RL agent that interacts with an environment, a 2D grid of dimensions 20*10: each (i,j) square in the grid gives out some reward to the agent when it visits that square. Each ...
0
votes
0answers
21 views

Relative Weighting of Loss Weights for Self-Play Reinforcement Learning

I am training some self play reinforcement learning agents to play 2 player board games like Connect 4, Othello, and The Game of the Amazons. For each game, there is a single neural network with 2 ...
0
votes
0answers
38 views

How to use Deep Q-Network with two-dimensional input? Hands-on Machine Learning 2

I'm studying with the book Hands-on Machine Learning with Scikit-Learn, Keras and TensorFlow, and I'm trying to implement the Deep Q-Network example that can be found on Github but that the input ...
0
votes
0answers
61 views

Are there any reinforcement learning benchmarks where the optimal policy is known for each environment?

There are multiple reinforcement learning (RL) benchmarks (i.e. a set of environments where we can test our RL algorithms), for example, the DeepMind Control Suite. However, given that I am currently ...
0
votes
0answers
51 views

mujoco environment in openai gym: observation and action explanation and control

I am new to RL and mujoco. I just set up mujoco and am testing the FetchPickAndPlace environment. I called the following methods: env.action_space.sample() returns ...
0
votes
0answers
44 views

Variance of the Gaussian policy is not decreasing while training the agent using Soft Actor-Critic method

I've written my own version of SAC(v2) for a problem with continuous action space. While training, the losses for the value network and both q functions steadily decrease down to 0.02-0.03. The loss ...
0
votes
0answers
19 views

How to design the step() method in a custom-built environment when the action space is a 2-dimentional matrix?

I have an action space that is a matrix. I am struggling with finding how to choose how to take the action when the number of possible actions is huge.
0
votes
1answer
44 views

How to let the agent choose how to populate a state space matrix in RL (using python)

I have an agent (drone) that has to allocate subchannels for different types of User Equipment. I have represented the subchannel allocation with a 2-dimentional binary matrix, that is initialized to ...
0
votes
0answers
23 views

Reinforcement comparison optimality

The following is definition of reinforcement comparison, which updates an average reward and a preference for each action http://incompleteideas.net/book/first/ebook/node22.html I want to know if this ...
0
votes
1answer
35 views

Why would the reward of A3C with LSTM suddenly drop off after many episodes?

I am training an A3C with stacked LSTM. During initial training, my model was giving descent +ve reward. However, after many episodes, its reward just goes to zero and is continuing for a long time. ...
0
votes
0answers
46 views

How to prove variance infinite of monte carlo ordinary importance sampling estimator

In example 5.5 of Sutton and Barto's book for proving infinite variance of first visit monte carlo ordinary importance sampling estimator, $\mathbb{E}[(\Pi_t\frac{\pi(A_t|S_t)}{b(A_t|S_t)}G_0)^2]$ is ...
0
votes
0answers
35 views

Reward and loss follow the same shape in DQN

If the accumulated reward increases, the loss increases and vice versa. This is a strange behaviour. See the figure below for an example. What is the possibility of having this behaviour in DQN? I ...
0
votes
0answers
48 views

Does Multi-Agent Deep Deterministic Policy Gradient also work with discrete action spaces?

I would like to ask if Multi-Agent Deep Deterministic Policy Gradient (MADDPG) works fine with discrete action space. DDPG works only with continuous action space, but I have read that MADDPG can also ...
0
votes
0answers
25 views

How does the second phase of the evaluation function described in this article work?

I am trying to create an evaluation function for a general game player based on the research from this article An Automatically-Generated Evaluation Function in General Game Playing. I can't ...
0
votes
0answers
27 views

Does normal A3C works well for continuous state space?

I am trying to create an A3C but it is giving same action for all the states during the training. The same action is also not an obvious way to maximise the reward. However I donot know if normal a3c ...
0
votes
0answers
46 views

Generalized advantage estimation DRL PPO

I have a question about generalized advantage estimation in multi agent environment. According to the formula I've computed GAE in this way: ...

1
9 10
11
12 13