Questions tagged [reinforcement-learning]
For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.
2,198
questions
0
votes
0
answers
3
views
Lunar Lander in Multi-agent environment
I'm trying to build a multi-agent lunar lander environment for the project and thought I could make it work because I read research paper on petting zoo library, but the problem is that as I try to ...
0
votes
0
answers
11
views
Building a OpenAI gym environment for a continuous resource management simulation
I am building an OpenAI Gym environment for a proprietary simulator. It simulates a number of servers (currently a fixed number; maybe variable in future) which are allocated virtualized resources ...
0
votes
0
answers
22
views
Reinforcement Learning - Independence between current state and future state
I'm working on a real problem with continuous large actions range, where my agent takes actions based only on the current state of the environment, transitioning to a future state that is unrelated to ...
0
votes
0
answers
8
views
Is model-based RL better suited for domain shift then model-free RL?
My intuition is that richer representations can be used for a larger number of downstream tasks and that model-based RL is more suited to produce such representations. Is there empirical work that ...
- 156
0
votes
0
answers
25
views
DDQN Snake keeps crashing into the wall
Edit:
I managed to fix this by changing the optimizer to SGD.
I am very new to reinforcement learning, and I attempted to create a DDQN for the game snake but for some reason it keeps learning to ...
0
votes
0
answers
36
views
My neural network does not appear to learn DQN [closed]
Good afternoon,
I'm trying to implement a DQN with Bellman Equations, however I don't know why but every prediction seems random. Just below here is my code.
My parameters :
...
0
votes
0
answers
21
views
Alphago zero test phase
I have started to comprehend alphago zero algorithm. We used MCTS to adjust the network parameters. After convergence in test phase do we use MCTS again??
0
votes
0
answers
20
views
Reduce Variance for Reinforce Algorithm
What strategies are there to reduce the variance of the policy gradient estimator of the REINFORCE algorithm? I know one possibility is to subtract a baseline as a running average of rewards from past ...
1
vote
1
answer
40
views
Using TD algorithms, if the value function of terminal states is always 0, why would a policy ever choose it?
Temporal difference algorithms (TD($\lambda$)) are tabular solutions to reinforcement learning problems. That is, they create a table of all the states in the problem, and estimate the expected long-...
- 148
0
votes
1
answer
21
views
What does the figure in Q-learning vs Expected SARSA actually show?
I might be blind.
But I wasn't able to find or figure out what the small difference between Q-learn and SARSA depicts in the following image;
(src).
What does the semi-circle show? and what does the ...
- 157
1
vote
1
answer
17
views
Train Deep Q-Learning Network on a game without source code
So I have some games that I like, and I'd like to create a net that can play them, just for fun. But I don't have their source code, so I can't just pull the information I want and create a state from ...
- 13
0
votes
0
answers
23
views
Inferring a transition model from incomplete data subject to rules
Consider a queue with number of items at time $t$ given by $x_t$. The number of items at time $t+1$ is determined probabilistically according to an unknown (and potentially non-stationary) transition ...
- 101
0
votes
0
answers
21
views
Embedding layers/entities in openAI's Hide and seek paper
I've recently come across a youtube video about openAI's hide and seek paper (https://openai.com/blog/emergent-tool-use/) and got really fascinated about the paper itself. But as I digging in the ...
1
vote
0
answers
16
views
Intelligent Agents with Personality - Implementation
I want to experiment to test the impact of agents' personalities in a team of agents on solving a given problem(for example, pathfinding). For example, one group will consist of two introverts and the ...
- 11
3
votes
2
answers
68
views
What is the definition of a continuous state/action space?
This question is a result of a discussion with one of my more math-minded friends. When I accidentally mentioned the term continuous state space, he corrected me by saying that I am most probably ...
- 67
1
vote
0
answers
16
views
Where does the proximal policy optimization objective's ratio term come from?
I will use the notation used in the proximal policy optimization paper.
What approximation is needed to arrive at the surrogate objective (equation (6) above) with the ratio $r_t(\theta)$?
Put ...
- 193
0
votes
0
answers
6
views
Replay Buffer taking long time to construct (Reinforcement Learning DQN with tf-agetns)
I'm new to Reinforcement Learning and I have some question. I am actually training some DQN using the tf-agents from tensorflow. And I recently learned that it's not possible to train a DQN using ...
- 1
0
votes
0
answers
34
views
How does the discount factor (gamma) and memory replay work in Deep Q-learning?
Maybe in over my head about this but I'm having a hard time understanding the discount factor in Deep Q-learning.
Correct me if I'm wrong (1):
To train a Deep Q-learning network, every N:th step of ...
- 1
0
votes
0
answers
6
views
Understanding the term "information leak" used in "Imitation" library
In this page Limitations on horizon length from the Imitation library, its stated (paraprhased) that the autors of the library recommend that the user sticks to fixed horizon length experiments, and ...
0
votes
0
answers
21
views
Calculating Curiosity with Friston's Free Energy in Reinforcement Learning
I'm trying to implement the paper A Curiosity Algorithm for Robots Based on the Free Energy Principle in a reinforcement learning environment using PyTorch, but I am unclear how curiosity is ...
0
votes
0
answers
31
views
How to normalize a observation (state-vector) which is a mix of different types
I want to use an RL (DQN or PPO) for my use case. The use case is actually simple: The agent should change the behavior of a person. The person is busy with some tasks like reading a book or listening ...
0
votes
1
answer
25
views
DQN Loss Function that does not have Ground Truth [closed]
I'm new to Reinforcement Learning and now I am trying to implement a Deep Q Network (DQN). I was going over this tutorial: https://towardsdatascience.com/reinforcement-learning-with-tensorflow-agents-...
- 1
0
votes
1
answer
42
views
Are there better loss functions than MSE for maze solver using deep learning?
I am a newbie in reinforcement learning, and I was doing a project on solving an agent maze solver using deep Q Learning. Currently, I am using the MSE loss function, but the agent is very slow or ...
- 1
0
votes
0
answers
21
views
How to evaluate the performance of off-line & model-free reinforcement leaning?
I'm currently studying on off-line reinforcement learning (RL) and trying to utilize it for medical data. Because it seemed hard to develop well-performing environment model, I decided to adopt model-...
- 1
0
votes
1
answer
45
views
Ways to improve DQN model learning snake?
I'm training a Deep Q-learning model on a snake game and I would like some ideas on how to improve the model and maybe also efficiency of training it.
The game is currently set to a 12x12 grid, a blue ...
- 1
1
vote
1
answer
22
views
Reinforcement Learning: sampled actions in k-armed bandit problem
I am reading the book Reinforcement Learning: An Introduction. Second edition (Richard S. Sutton and Andrew G. Barto). In the k-armed bandit problem using $\varepsilon$-greedy selection method, the ...
- 13
0
votes
0
answers
11
views
How can I keep markov property when controlling many agents?
I am working on a project in which I am training a multiagent system to find a minimum in a scalar field. I have many agents that will receive information about the position of some of the other ...
1
vote
0
answers
33
views
Would the optimal policy remain same, if I replace R with V*?
In the context of RL, say I'm performing Value Iteration on a reward function R1. And the converged optimal policy is P1 and values are V1.
Then, let's say I set rewards to be R2=V1 and perform value ...
1
vote
1
answer
21
views
Optimality of Policy Iteration
In reinforcement learning, what guarantees that policy iteration would find the globally optimal solution and not just any local maximum?
I'm reading the book "Reinforcement Learning: An ...
- 233
0
votes
1
answer
32
views
How do you apply Q-learning when there are too many possible actions?
When the number of states in the Q-learning is large, we can refer to approximate Q-learning , but what should we do when we have a large number of actions?
- 1
2
votes
1
answer
40
views
Why $V^{\pi^*}(s) = \max_{a \in A}Q^{\pi^*}(s,a),\forall s \in S$ in reinforcement learning?
When i read some notes about RL i encounterd following equation and try to prove it:
$$
V^{\pi^*}(s) = \max_{a \in A}Q^{\pi^*}(s, a),\forall s \in S
$$
Here is my attemption:
Firstly, i only need to ...
- 21
0
votes
1
answer
41
views
OpeanAI Gym. Train problem: invalid values
I have a problem with my reinforcement learning model.
I am trying to simulate an electric battery storage. To keep it as simple as possible, the efficiency of charge, storage and discharge are 100%. ...
- 3
0
votes
0
answers
24
views
Reinforcement learning PPO-clip agent returning softmax prediction of 1
I have discrete action space with 3 actions. I use distributed PPO-clip algorithm with these hyperparameters:
Workers: 32
Optimizer: Adam
Learning rate: 0.000005
Epochs: 20
Batch size: 256
Episode ...
- 1
0
votes
1
answer
38
views
ML-based algorithm/software for solving a sudoku puzzle a human way
I am a new contributor and have no experience in ML, so this first question is a general one.
I've developed a sudoku solving app and since then I wonder whether it would be feasible to design a ML-...
1
vote
1
answer
41
views
What is the difference between these two versions of the Bellman equation?
The first version is the one I am most familiar with:
$$V_\pi(s) = \sum_{a}^{}\pi(a|s) \sum_{s'}^{}T(s, a, s')[R(s, a, s') + \gamma V_\pi(s')]$$
where $T(s, a, s')$ represents the probability of ...
- 123
-1
votes
3
answers
92
views
Why are neural networks used as reinforcement learning model value functions? [closed]
My understanding is that a value function in reinforcement learning returns a value that represents how "good" it is to be in a given state. How does a network, such as the network in this ...
- 1
0
votes
1
answer
22
views
Clarification on notations of the theorems in conservative q-learning paper
In Theorem 3.1 of the conservative q-learning paper, what is the meaning of
$ (I - \gamma P^{\pi})^{-1} [\frac{\mu(a|s)}{\hat \pi_{\beta}(a|s)}](s, a)$ ? I thought
$(I - \gamma P^{\pi})^{-1}$ is to be ...
- 58
0
votes
0
answers
12
views
Is there a benefit to grouping observation attributes in spaces.Dict [closed]
I have an observation object with different array attributes, e.g.
attribute1 = [...]
attribute2 = [...]
attribute3 = [...]
.
.
.
Some of these ...
0
votes
0
answers
10
views
how to handle different objectives in Atari games in reinforcement learning
My impression on DeepMind's Deep-Q RL for learning Atari games paper is that it uses the same model to learn to play multiple different games at the same time. I wonder how did the RL agent learn in ...
- 113
1
vote
1
answer
52
views
Neural Network output for the game of Checkers
I'm trying to train a RL agent to play the game of checkers (AlphaZero style) and so far I've managed a proof of concept training a connect 4 agent up until perfection. However, unlike connect 4, ...
- 25
0
votes
0
answers
14
views
How to normalize input data to Reinforcement learning platform (Gym and stable-baselines)
I created a custom environment with Gym and trained it with stable baseline 3 algorithms.
The observation and space action are both continues.
The observation space includes 10 values and action space ...
- 1
0
votes
0
answers
14
views
Devise a model/Goal Based Agent for simple Pacman Game
I have to program simple pacman game in figure below that consisting of 4*4 grid (not GUI based).
Explaination
The starting point of pacman is cell 0 and its goal is to consume/eat maximum food ...
0
votes
0
answers
20
views
What reward should be selected for transition states to make the agent reach the terminal state (destination) faster? negative, positive, or zero?
Consider the simple environment below, where the gray cells are the terminal states and the agent receives a reward of $-5$ for taking any action in these states. The nonterminal states are $S = \{1, ...
- 11
0
votes
1
answer
41
views
What makes TRPO an Actor-critic method? Where is the critic?
From what I understand, Trust Region Policy Optimization (TRPO) is a modification on Natural Policy Gradient (NPG) that derives the optimal step size $\beta$ from a KL constraint between the new and ...
- 141
2
votes
2
answers
56
views
Why clip the PPO objective on only one side?
In PPO with clipped surrogate objective (see the paper here), we have the following objective:
The shape of the function is shown in the image below, and depends on whether the advantage is positive ...
- 21
0
votes
0
answers
19
views
Action space for a single agent environment with multiple actions
How to define an action space in a gym environment where the agent´s output is a tensor of shape [1,25]?
I am working on Travelling salesman problem using DRL where the NN(agent) output`s the sequence ...
0
votes
0
answers
16
views
What is an example of an *optimal* stochastic policy that assigns a nonzero probability to an action with a lower expected value?
A stochastic policy means that an agent has probabilities of choosing their available actions, given a state: $\pi(a|s)$.
However in an optimal stochastic policy for a given state, you would assume ...
- 123
0
votes
0
answers
11
views
reinforcement learning distributional task and reward
just want to ask in RL, if the task is distributional, like T ~ p(T), task T is a belongs to a distribution, in this case, does the reward functions for each sampled task Ti are in different form ? ...
- 1
1
vote
0
answers
40
views
MDP with a non-markovian reward function?
I have set up a RL environment and it converges to a decent solution when using a reward function:
$R(s_t,a_t) = fenv(s_t, a_t)$ , where $fenv$ is the environment dynamics.
Now, i want to change the ...
- 11
0
votes
0
answers
12
views
Instruction-based language model without neural network
What kind of technology could be used to develop an AI with the same use case as InstructGPT (i.e. generating text from an instruction) but without using neural networks?
Obviously the performance ...
user62498