Questions tagged [reinforcement-learning]

For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.

Filter by
Sorted by
Tagged with
0 votes
0 answers
3 views

Lunar Lander in Multi-agent environment

I'm trying to build a multi-agent lunar lander environment for the project and thought I could make it work because I read research paper on petting zoo library, but the problem is that as I try to ...
0 votes
0 answers
11 views

Building a OpenAI gym environment for a continuous resource management simulation

I am building an OpenAI Gym environment for a proprietary simulator. It simulates a number of servers (currently a fixed number; maybe variable in future) which are allocated virtualized resources ...
0 votes
0 answers
22 views

Reinforcement Learning - Independence between current state and future state

I'm working on a real problem with continuous large actions range, where my agent takes actions based only on the current state of the environment, transitioning to a future state that is unrelated to ...
0 votes
0 answers
8 views

Is model-based RL better suited for domain shift then model-free RL?

My intuition is that richer representations can be used for a larger number of downstream tasks and that model-based RL is more suited to produce such representations. Is there empirical work that ...
0 votes
0 answers
25 views

DDQN Snake keeps crashing into the wall

Edit: I managed to fix this by changing the optimizer to SGD. I am very new to reinforcement learning, and I attempted to create a DDQN for the game snake but for some reason it keeps learning to ...
0 votes
0 answers
36 views

My neural network does not appear to learn DQN [closed]

Good afternoon, I'm trying to implement a DQN with Bellman Equations, however I don't know why but every prediction seems random. Just below here is my code. My parameters : ...
0 votes
0 answers
21 views

Alphago zero test phase

I have started to comprehend alphago zero algorithm. We used MCTS to adjust the network parameters. After convergence in test phase do we use MCTS again??
0 votes
0 answers
20 views

Reduce Variance for Reinforce Algorithm

What strategies are there to reduce the variance of the policy gradient estimator of the REINFORCE algorithm? I know one possibility is to subtract a baseline as a running average of rewards from past ...
1 vote
1 answer
40 views

Using TD algorithms, if the value function of terminal states is always 0, why would a policy ever choose it?

Temporal difference algorithms (TD($\lambda$)) are tabular solutions to reinforcement learning problems. That is, they create a table of all the states in the problem, and estimate the expected long-...
0 votes
1 answer
21 views

What does the figure in Q-learning vs Expected SARSA actually show?

I might be blind. But I wasn't able to find or figure out what the small difference between Q-learn and SARSA depicts in the following image; (src). What does the semi-circle show? and what does the ...
1 vote
1 answer
17 views

Train Deep Q-Learning Network on a game without source code

So I have some games that I like, and I'd like to create a net that can play them, just for fun. But I don't have their source code, so I can't just pull the information I want and create a state from ...
  • 13
0 votes
0 answers
23 views

Inferring a transition model from incomplete data subject to rules

Consider a queue with number of items at time $t$ given by $x_t$. The number of items at time $t+1$ is determined probabilistically according to an unknown (and potentially non-stationary) transition ...
  • 101
0 votes
0 answers
21 views

Embedding layers/entities in openAI's Hide and seek paper

I've recently come across a youtube video about openAI's hide and seek paper (https://openai.com/blog/emergent-tool-use/) and got really fascinated about the paper itself. But as I digging in the ...
1 vote
0 answers
16 views

Intelligent Agents with Personality - Implementation

I want to experiment to test the impact of agents' personalities in a team of agents on solving a given problem(for example, pathfinding). For example, one group will consist of two introverts and the ...
  • 11
3 votes
2 answers
68 views

What is the definition of a continuous state/action space?

This question is a result of a discussion with one of my more math-minded friends. When I accidentally mentioned the term continuous state space, he corrected me by saying that I am most probably ...
  • 67
1 vote
0 answers
16 views

Where does the proximal policy optimization objective's ratio term come from?

I will use the notation used in the proximal policy optimization paper. What approximation is needed to arrive at the surrogate objective (equation (6) above) with the ratio $r_t(\theta)$? Put ...
  • 193
0 votes
0 answers
6 views

Replay Buffer taking long time to construct (Reinforcement Learning DQN with tf-agetns)

I'm new to Reinforcement Learning and I have some question. I am actually training some DQN using the tf-agents from tensorflow. And I recently learned that it's not possible to train a DQN using ...
0 votes
0 answers
34 views

How does the discount factor (gamma) and memory replay work in Deep Q-learning?

Maybe in over my head about this but I'm having a hard time understanding the discount factor in Deep Q-learning. Correct me if I'm wrong (1): To train a Deep Q-learning network, every N:th step of ...
  • 1
0 votes
0 answers
6 views

Understanding the term "information leak" used in "Imitation" library

In this page Limitations on horizon length from the Imitation library, its stated (paraprhased) that the autors of the library recommend that the user sticks to fixed horizon length experiments, and ...
0 votes
0 answers
21 views

Calculating Curiosity with Friston's Free Energy in Reinforcement Learning

I'm trying to implement the paper A Curiosity Algorithm for Robots Based on the Free Energy Principle in a reinforcement learning environment using PyTorch, but I am unclear how curiosity is ...
0 votes
0 answers
31 views

How to normalize a observation (state-vector) which is a mix of different types

I want to use an RL (DQN or PPO) for my use case. The use case is actually simple: The agent should change the behavior of a person. The person is busy with some tasks like reading a book or listening ...
0 votes
1 answer
25 views

DQN Loss Function that does not have Ground Truth [closed]

I'm new to Reinforcement Learning and now I am trying to implement a Deep Q Network (DQN). I was going over this tutorial: https://towardsdatascience.com/reinforcement-learning-with-tensorflow-agents-...
0 votes
1 answer
42 views

Are there better loss functions than MSE for maze solver using deep learning?

I am a newbie in reinforcement learning, and I was doing a project on solving an agent maze solver using deep Q Learning. Currently, I am using the MSE loss function, but the agent is very slow or ...
  • 1
0 votes
0 answers
21 views

How to evaluate the performance of off-line & model-free reinforcement leaning?

I'm currently studying on off-line reinforcement learning (RL) and trying to utilize it for medical data. Because it seemed hard to develop well-performing environment model, I decided to adopt model-...
0 votes
1 answer
45 views

Ways to improve DQN model learning snake?

I'm training a Deep Q-learning model on a snake game and I would like some ideas on how to improve the model and maybe also efficiency of training it. The game is currently set to a 12x12 grid, a blue ...
  • 1
1 vote
1 answer
22 views

Reinforcement Learning: sampled actions in k-armed bandit problem

I am reading the book Reinforcement Learning: An Introduction. Second edition (Richard S. Sutton and Andrew G. Barto). In the k-armed bandit problem using $\varepsilon$-greedy selection method, the ...
0 votes
0 answers
11 views

How can I keep markov property when controlling many agents?

I am working on a project in which I am training a multiagent system to find a minimum in a scalar field. I have many agents that will receive information about the position of some of the other ...
1 vote
0 answers
33 views

Would the optimal policy remain same, if I replace R with V*?

In the context of RL, say I'm performing Value Iteration on a reward function R1. And the converged optimal policy is P1 and values are V1. Then, let's say I set rewards to be R2=V1 and perform value ...
1 vote
1 answer
21 views

Optimality of Policy Iteration

In reinforcement learning, what guarantees that policy iteration would find the globally optimal solution and not just any local maximum? I'm reading the book "Reinforcement Learning: An ...
0 votes
1 answer
32 views

How do you apply Q-learning when there are too many possible actions?

When the number of states in the Q-learning is large, we can refer to approximate Q-learning , but what should we do when we have a large number of actions?
  • 1
2 votes
1 answer
40 views

Why $V^{\pi^*}(s) = \max_{a \in A}Q^{\pi^*}(s,a),\forall s \in S$ in reinforcement learning?

When i read some notes about RL i encounterd following equation and try to prove it: $$ V^{\pi^*}(s) = \max_{a \in A}Q^{\pi^*}(s, a),\forall s \in S $$ Here is my attemption: Firstly, i only need to ...
  • 21
0 votes
1 answer
41 views

OpeanAI Gym. Train problem: invalid values

I have a problem with my reinforcement learning model. I am trying to simulate an electric battery storage. To keep it as simple as possible, the efficiency of charge, storage and discharge are 100%. ...
  • 3
0 votes
0 answers
24 views

Reinforcement learning PPO-clip agent returning softmax prediction of 1

I have discrete action space with 3 actions. I use distributed PPO-clip algorithm with these hyperparameters: Workers: 32 Optimizer: Adam Learning rate: 0.000005 Epochs: 20 Batch size: 256 Episode ...
  • 1
0 votes
1 answer
38 views

ML-based algorithm/software for solving a sudoku puzzle a human way

I am a new contributor and have no experience in ML, so this first question is a general one. I've developed a sudoku solving app and since then I wonder whether it would be feasible to design a ML-...
1 vote
1 answer
41 views

What is the difference between these two versions of the Bellman equation?

The first version is the one I am most familiar with: $$V_\pi(s) = \sum_{a}^{}\pi(a|s) \sum_{s'}^{}T(s, a, s')[R(s, a, s') + \gamma V_\pi(s')]$$ where $T(s, a, s')$ represents the probability of ...
  • 123
-1 votes
3 answers
92 views

Why are neural networks used as reinforcement learning model value functions? [closed]

My understanding is that a value function in reinforcement learning returns a value that represents how "good" it is to be in a given state. How does a network, such as the network in this ...
0 votes
1 answer
22 views

Clarification on notations of the theorems in conservative q-learning paper

In Theorem 3.1 of the conservative q-learning paper, what is the meaning of $ (I - \gamma P^{\pi})^{-1} [\frac{\mu(a|s)}{\hat \pi_{\beta}(a|s)}](s, a)$ ? I thought $(I - \gamma P^{\pi})^{-1}$ is to be ...
0 votes
0 answers
12 views

Is there a benefit to grouping observation attributes in spaces.Dict [closed]

I have an observation object with different array attributes, e.g. attribute1 = [...] attribute2 = [...] attribute3 = [...] . . . Some of these ...
0 votes
0 answers
10 views

how to handle different objectives in Atari games in reinforcement learning

My impression on DeepMind's Deep-Q RL for learning Atari games paper is that it uses the same model to learn to play multiple different games at the same time. I wonder how did the RL agent learn in ...
  • 113
1 vote
1 answer
52 views

Neural Network output for the game of Checkers

I'm trying to train a RL agent to play the game of checkers (AlphaZero style) and so far I've managed a proof of concept training a connect 4 agent up until perfection. However, unlike connect 4, ...
0 votes
0 answers
14 views

How to normalize input data to Reinforcement learning platform (Gym and stable-baselines)

I created a custom environment with Gym and trained it with stable baseline 3 algorithms. The observation and space action are both continues. The observation space includes 10 values and action space ...
0 votes
0 answers
14 views

Devise a model/Goal Based Agent for simple Pacman Game

I have to program simple pacman game in figure below that consisting of 4*4 grid (not GUI based). Explaination The starting point of pacman is cell 0 and its goal is to consume/eat maximum food ...
0 votes
0 answers
20 views

What reward should be selected for transition states to make the agent reach the terminal state (destination) faster? negative, positive, or zero?

Consider the simple environment below, where the gray cells are the terminal states and the agent receives a reward of $-5$ for taking any action in these states. The nonterminal states are $S = \{1, ...
  • 11
0 votes
1 answer
41 views

What makes TRPO an Actor-critic method? Where is the critic?

From what I understand, Trust Region Policy Optimization (TRPO) is a modification on Natural Policy Gradient (NPG) that derives the optimal step size $\beta$ from a KL constraint between the new and ...
2 votes
2 answers
56 views

Why clip the PPO objective on only one side?

In PPO with clipped surrogate objective (see the paper here), we have the following objective: The shape of the function is shown in the image below, and depends on whether the advantage is positive ...
  • 21
0 votes
0 answers
19 views

Action space for a single agent environment with multiple actions

How to define an action space in a gym environment where the agent´s output is a tensor of shape [1,25]? I am working on Travelling salesman problem using DRL where the NN(agent) output`s the sequence ...
0 votes
0 answers
16 views

What is an example of an *optimal* stochastic policy that assigns a nonzero probability to an action with a lower expected value?

A stochastic policy means that an agent has probabilities of choosing their available actions, given a state: $\pi(a|s)$. However in an optimal stochastic policy for a given state, you would assume ...
  • 123
0 votes
0 answers
11 views

reinforcement learning distributional task and reward

just want to ask in RL, if the task is distributional, like T ~ p(T), task T is a belongs to a distribution, in this case, does the reward functions for each sampled task Ti are in different form ? ...
  • 1
1 vote
0 answers
40 views

MDP with a non-markovian reward function?

I have set up a RL environment and it converges to a decent solution when using a reward function: $R(s_t,a_t) = fenv(s_t, a_t)$ , where $fenv$ is the environment dynamics. Now, i want to change the ...
0 votes
0 answers
12 views

Instruction-based language model without neural network

What kind of technology could be used to develop an AI with the same use case as InstructGPT (i.e. generating text from an instruction) but without using neural networks? Obviously the performance ...
user avatar

1
2 3 4 5
44