Questions tagged [deep-rl]

For questions related to deep reinforcement learning (DRL), that is, RL combined with deep learning. More precisely, deep neural networks are used to represent e.g. value functions or policies.

Filter by
Sorted by
Tagged with
0
votes
0answers
11 views

tfp.Distributions.Categorical.sample() is picking the same action everytime after certain iterations

I have written a code for an RL agent such that at each state the model calculates the probabilities of all possible actions and samples one action randomly to proceed further. To acheive this, I have ...
0
votes
0answers
19 views

FrozenLake-v0 not training using REINFORCE

I am implementing a simple REINFORCE (policy gradient) algorithm for openAI's FrozenLake-v0 environment. However, it does not seem to learn anything at all. I have used the same neural architecture ...
2
votes
0answers
51 views

Can Q-learning be used for my scenario, and how might I do so?

I have already asked 2-3 general questions w.r.t Q learning and now I am asking a scenario specific one. I will try to be concise and understandable. I really really need help. Scenario: I have a ...
-1
votes
1answer
36 views

Is it possible to train an RL agent using images?

I have an image which consists of a start and an end point, the journey has some obstacles which have to be avoided. Is it possible to train an RL agent using such images to find the best path ...
0
votes
1answer
30 views
0
votes
0answers
32 views

How to deal with Q-learning having low variance in predicted Q-values?

I have a neural network that takes the state (which contains a lot of data), and the possible action (which is very little data), and predicts the Q-value of the action. I am double Q-learning. I've ...
4
votes
0answers
488 views

Why isn't a target network used for the critic in on-policy actor-critic methods?

Based on my research, I've seen so many on-policy AC approaches that utilise a critic network to estimate the value function $V$. The Bellman equation for the value function is as bellow: $$ V_\pi(s_t)...
0
votes
0answers
15 views

How to fix loss?

I am trying to use the Vanilla Policy Gradient algorithm to solve MountainCar-v0. I have taken the code from pytorch repo. ...
4
votes
1answer
65 views

Why exactly was previously believed that the deterministic policy gradient did not exist?

I'm reading the paper Deterministic Policy Gradient Algorithms, David Silver et al. First of all, in the introduction, the author says that It was previously believed that the deterministic policy ...
2
votes
1answer
76 views

Reinforcement Learning method suitable for a large discrete action space with high sample efficiency

Consider the following problem. We have a process, that generates $N$ stones (e.g. 2000) in one batch $b$. Every pebble has state $s_{i}^b$ and reward $s_i^b$. After choosing one pebble $i$ from the $...
1
vote
1answer
38 views

Training a reinforcement learning agent that can decide to continue or end the game

I am trying to use reinforcement learning to let an agent learn simultaneously how to play a game and when to end a game. The task is to find a single target in a grid of locations. At each time step, ...
1
vote
0answers
18 views

Transferring a Q-learning policy to larger instances

How do I best transfer and fine-tune a Q-learning policy that was trained on small instances to large instances? Some more details on the problem: I am currently trying to derive a decision policy for ...
2
votes
1answer
52 views

How to encourage the reinforcement-learning agent to reach the goal as quickly as possible, and what's the effect of discount factor?

I am trying to use reinforcement learning to solve a task and compare its performance to humans. The task is to find a single target in a fixed number of locations. At each step, the agent will pick ...
0
votes
0answers
29 views

Should training be done in number of episodes or number of timesteps?

In the past few deep reinforcement learning projects I've worked with, a problem I have encountered is whether to frame training time in terms of number of episodes or number of timesteps. For ...
2
votes
1answer
36 views

Do you need a terminal state when using double deep q networks?

I just got my agent training, and I'm wondering if the terminal flags are necessary when sampling from the replay buffer. The game I'm implementing the agent in has two different ways the game can end,...
0
votes
1answer
44 views

Appropriate ML algorithm to solve a cutting pattern problem

I have a rectangular area, where I need to place some 2 dimensional geometrical shapes - like a square or circle or a little more complicated shapes. And after the arrangement these shapes should be ...
0
votes
1answer
43 views

Can a Reinforcement Learning problem with multiple simultaneous actions be formalized as a Multiagent Partially Observable Markov Decision Process?

Consider the following decision making problem. We have a controller that selects locations from a grid of coordinates and captures an image (observation $o_t$) with a camera at each location (action $...
1
vote
0answers
23 views

Reinforcement Learning for Finite Time Horizon and Non-Trivial Terminal Reward

I notice that most Deep Reinforcement Learning (DRL) works focus on Markov Decision Process (MDP) with an infinite time horizon. Are there any algorithms that work well on finite MDP and non-trivial ...
2
votes
1answer
38 views

Is it possible to apply a particular exploration policy for the on-policy RL agents?

Is it possible to use any particular strategy to explore (e.g. metaheuristics) in on-policy algorithms (e.g. in PPO) or is it only possible to define particular policies to explore in off-policy ...
1
vote
0answers
32 views

What deep reinforcement learning algorithm should I use for my problem?

So here is a description of my problem: Essentially, I have a large amount of files filled with code for a number of different tasks. However, lets say these codes are inefficient, and should be ...
0
votes
0answers
27 views

Why would the agent always take the same action in the test environment?

I'm trying to set up an agent with PPO2. But I've tried with: A2C DQN PPO2 However, in the test environment, the agent always takes the same action, and the total profit is negative. What can be the ...
0
votes
0answers
21 views

Joined vs Separate optimizer for Actor-Critic

Say that I have a simple Actor-Critic architecture, (I am not familiar with Tensorflow, but) in Pytorch we need to specify the parameters when defining an optimizer (SGD, Adam, etc) and therefore we ...
0
votes
0answers
17 views

How does the Markov assumption hold true for episodic task?

The Markov assumption assumes that the current state is sufficient for taking the next action. Consider an episodic task, where the RL agent is trying to learn to play the game of tic-tac-toe. Here, ...
1
vote
1answer
33 views

How are rewards calculated for episodic tasks like playing chess or tic-tac-toe?

I am new to Reinforcement Learning and trying to understand the concept of reaping rewards during episodic tasks. I think in games like tic-tac-toe, rewards will be in terms of a win or lose. But does ...
1
vote
0answers
33 views

Could we add clipping in the output layer of the actor in DDPG?

I have a doubt about how clipping affects the training of the RL agents. In particular, I have come across a code for training DDPG agents, the pseudo-code is the following: ...
1
vote
1answer
56 views

Multi Agent Deep Reinforcement Learning for continuous and discrete action

I am looking to have a cooperative multi agent reinforcement learning framework where one agent has a discrete action space and another agent has a continuous action space. Is there a way to do this ...
1
vote
0answers
32 views

Setting initial values in DDPG to favor better actions

I'm working on a problem using DDPG. Is it possible to add some intelligence in the initialization phase, such that the convergence time is improved/shortened and local optima are avoided as much as ...
0
votes
1answer
48 views

Are the Q-values of DQN bounded at a single timestep?

Consider that we have an agent that has a set of thousands of different actions at each timestep. The reward function in $R:S \rightarrow\{0,1\}$. Let $Q_{t}^\pi(s,a)$ be the estimate from the neural ...
1
vote
2answers
74 views

Should I apply normalization to the observations in deep reinforcement learning?

I am new to DRL and trying to implement my custom environment. I want to know if normalization and regularization techniques are as important in RL as in Deep Learning. In my custom environment, the ...
1
vote
0answers
35 views

How to parallelize multi-agent DDPG (MADDPG)

I am experimenting with MADDPG algorithm implemented in this repo. Since there were only a few agents (2-3) in the implementation (also in the original paper) steps like parameter updates, action ...
2
votes
2answers
159 views

What is the optimal score for Tic Tac Toe for a reinforcement learning agent against a random opponent?

I guess this problem is encountered by everyone trying to solve Tic Tac Toe with various flavors of reinforcement learning. The answer is not "always win" because the random opponent may ...
1
vote
1answer
20 views

How to embed game grid state with walls as an input to neural network

I've read most of the posts on here regarding this subject, however most of them deal with gameboards where there are two different categories of single pieces on a board without walls etc. My game ...
1
vote
1answer
46 views

PPO when does the update happen?

In many places, it says PPO and Actor-Critic methods in general use TD-updates, but in the loss function for PPO, the Value function loss component uses the difference between output of the value ...
1
vote
1answer
60 views

Implementing Multiple NNs in one DQN model?

I'm trying to build a DQN Agent to take a set of 10 best actions simultaneously (integer values from 1 to 100) as outputs per episode. The input is a float. The goal is to find the optimal combination ...
1
vote
1answer
58 views

When to activate batch normalization and dropout in deep Q-learning?

In the vanilla version of deep Q-learning, there are three places where the Q-network is queried: When exploring. When training: a. When calculating the optimal value of the state reached by an ...
2
votes
1answer
32 views

Is Reinforcement Learning capable of learning complex functions (such as producing a 3d model given an image)?

I want to build an AI that can convert an image of a subject into an anatomically accurate 3D model. To do this, I was thinking of adapting the following code for Deep Deterministic Policy Gradient: ...
1
vote
0answers
54 views

In what situation would you want to use NEAT over reinforcement learning?

NEAT is an evolutionary algorithm. When would you want to use NEAT over more traditional/common RL algorithms like PPO or SAC etc. What advantage does it give you?
0
votes
0answers
32 views

Is it possible to use Softmax as an activation function for actor (policy) network in TD3 or SAC Reinforcement learning algorithms?

As I understand from literature, normally, the last activation in an actor (policy) network in TD3 and SAC algorithms is a Tanh function, which is scaled by a certain limit. My action vector is ...
1
vote
1answer
35 views

In deep reinforcement learning, what is this model with state as input and value as output?

I was looking at this implementation for creating an agent for playing Tetris using DeepRL. This model uses "a state based on the statistics of the board after a potential action. All predictions ...
0
votes
1answer
93 views

Minimizing trial and error in reinforcement learning

The initial environment state is 0.25. Each time step the agent performs a discrete action of 0 or ...
0
votes
1answer
78 views

Deep Q-Learning with multiple discrete actions

I am working on a DQN project with Pytorch, where I should choose multiple discrete actions, each in a range, say, (0, 15). I am wondering how I can model it, such ...
0
votes
0answers
19 views

Reinforcement learning: better performance with a stochastic environment

I modified the cartpole environment with one line to make it stochastic: ...
0
votes
0answers
17 views

Why use two different embeddings for actions in this paper?

I was reading this paper Top-𝐾 Off-Policy Correction for a REINFORCE Recommender System and I'm wondering is there a particular advantage to use different embeddings for actions, one embedding is ...
0
votes
0answers
21 views

RL agent policy performs worse than random policy

I am training a trading bot with TD3 and SAC algorithms. During the first 10k steps it takes uniformly random actions before running policy learnt so far. The agent starts to do gradient descent ...
1
vote
1answer
53 views

How to use a heuristic policy to increase sample efficiency of a deep reinforcement learning agent?

I have a heuristic solution to a problem which works quite well when certain environmental parameters are known and unchanging. However, in a real world setting these parameters will not be known and ...
2
votes
1answer
192 views

What do equations 1 and 3 describe in the "Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels" paper?

This paper uses image augmentation to improve RL algorithms. It contains the following paragraph - "Our approach, DrQ, is the union of the three separate regularization mechanisms introduced ...
0
votes
0answers
32 views

PPO: decreasing rewards as steps increase

How to explain this? Is this normal? Does this mean a bad design of the environment?
2
votes
0answers
44 views

How to prove Lemma 1.6 in the book "Reinforcement Learning: Theory and Algorithms"

I am trying to prove the following lemma from Reinforcement Learning: Theory and Algorithms on page 8. Lemma 1.6. We have that: $$ \left[(1-\gamma)\left(I-\gamma P^{\pi}\right)^{-1}\right]_{(s, a),\...
0
votes
1answer
32 views

Where can I find the original conference paper that introduced Q-learning and Deep Q-Learning?

I tried searching a lot, but I could neither find the paper that introduced Q-Learning nor the paper that introduced Deep Q Learning. If anyone knows anything about it please do tell me.
1
vote
1answer
36 views

How to interpret the policy gradient expression in reinforcement learning?

I'm currently going through the OpenAI's spinning up introduction course to reinforcement learning. On one of the final sections, they derive an expression for the gradient of the undiscounted return ...

1
2 3 4 5
7