Questions tagged [reinforcement-learning]

For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.

604 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
1
vote
0answers
83 views

Designing Policy-Network for Deep-RL with Large, Variable Action Space

I am attempting a project involving training an agent to play a game using deep reinforcement learning. This project has a few features that complicate the design of the neural network: The action ...
1
vote
0answers
23 views

Understanding example for Improved Policy Iteration for POMDPs

I was going through this paper by Hansen. This paper proposes policy improvement by first converting set of $\alpha$ vectors into finite state controller and then comparing them to obtain improved ...
1
vote
0answers
39 views

DDQN Agent in Othello (Reversi) game struggle to learn

This is my first question on this forum and I would like to welcome everyone. I am trying to implement DDQN Agent playing Othello (Reversi) game. I have tried multiple things but the agent which seems ...
1
vote
0answers
64 views

What kind of reinforcement learning method does AlphaGo Deepmind use to beat the best human Go player?

In reinforcement learning, there are model-based versus model-free methods. Within model-based ones, there are policy-based and value-based methods. AlphaGo Deepmind RL model has beaten the best Go ...
1
vote
0answers
70 views

How to deal with KerasRL DDPG algorithm getting stuck in a local optima?

I am using KerasRL DDPG to try to learn a policy on my own custom environment, but the agent is stuck in a local optima although I am adding the OrnsteinUhlenbeck randomization process. I used the ...
1
vote
0answers
65 views

Stack of Planes as the Action Space Representation for AlphaZero (Chess)

I have a question regarding the action space of the policy network used in AlphaZero. From the paper: We represent the policy π(a|s) by a 8 × 8 × 73 stack of planes encoding a probability ...
1
vote
1answer
85 views

How to deal with the time delay in reinforcement learning?

I have a question regarding the time delay in reinforcement learning (RL). In the RL, one has state, reward and action. It is usually assumed that (as far as I understand it) when the action is ...
1
vote
0answers
83 views

What are the differences between Proximal Policy Optimization versions PPO1 and PPO2?

When Proximal Policy Optimization (PPO) was released, it was accompanied by a paper describing it. Later, the authors at OpenAI introduced a second version of PPO, called PPO2 (whereas the original ...
1
vote
0answers
16 views

Would it make sense to share the layers (except the last one) of the neural networks in Double DQN?

Context: Double Q-learning was introduced to prevent the maximization bias from q-learning. Instead of learning a single Q-network, we can learn two (or in general $K > 1$) and our Q-estimate would ...
1
vote
0answers
13 views

Are the relative magnitudes of the learned and optimal state value function the same?

I have been reading recently about value and policy iteration. I tried to code the algorithms to understand them better and in the process I discovered something and I am not sure why is the case (or ...
1
vote
0answers
40 views

Why is the ideal exploration parameter in the UCT algorithm $\sqrt{2}$?

From Wikipedia, in the Monte-Carlo Tree Search algorithm, you should choose the node that maximizes the value: $${\displaystyle {\frac {w_{i}}{n_{i}}}+c{\sqrt {\frac {\ln N_{i}}{n_{i}}}}},$$ where ${...
1
vote
1answer
41 views

Aside from specific training sets, what distinguishes the capabilities of different AI implementations?

(Disclaimer: I don't know much about ML/AI, besides some basic ideas behind it all.) It seems like ML/AI models can often be boiled down to statistics, where certain levers (weights) get fine-tuned ...
1
vote
0answers
16 views

Looking for a good approach for building an automated director for a racing game spectator mode

I'm building a tool that should assist a director to broadcast a racing game. I want this tool to suggest the human director which car to focus on and with which camera (among the available ones). I ...
1
vote
0answers
29 views

How to compute the Retrace target for multi-step off-policy Reinforcement Learning?

I am implementing the A3C algorithm and I want to add off-policy training using Retrace but I am having some trouble understanding how to compute the retrace target. Retrace is used in combination ...
1
vote
0answers
27 views

Multi-armed bandit problem without getting rewards

In a 2-armed-bandit problem, an agent has an opportunity to see n reward for each action. Now the agent should choose actions m times and maximize the expected reward in these m decisions. but it cant ...
1
vote
0answers
49 views

Why does PPO lead to a worse performance than TRPO in the same task?

I am training an agent with an Actor-Critic network and update it with TRPO so far. Now, I tried out PPO and the results are drastically different and bad. I only changed from TRPO to PPO, the rest of ...
1
vote
0answers
43 views

How exactly is hindsight experience replay related to potential-based reward shaping?

One of the reviewers of the HER paper (which was accepted as a NIPS conference paper) wrote Overall, I'd say that it's not a huge/deep idea, but a very nice addition to the learning toolbox. When it ...
1
vote
0answers
34 views

How to use unmodified input in neural network?

My question may be a bit hard to explain... My neural network learns a categorical distribution, which serves as an index. This index will look up the value (= action_mean) in Input 2. From this ...
1
vote
0answers
71 views

Does the order in which the features are concatenated to create the state (or observation) matter?

I'm experimenting with an RL agent that interacts with the following environment. The learning algorithm is double DQN. The neural network represents the function from state to action. It's build with ...
1
vote
0answers
56 views

What adapts an algorithm to continuous or to discrete action spaces?

Some RL algorithms can only be used for environments with continuous action spaces (e.g TD3, SAC), while others only for discrete action spaces (DQN), and some for both REINFORCE and other policy ...
1
vote
0answers
40 views

Where can I find pre-trained agents able to play games with multiple stages like exploration, dialog, combat?

My goal is to create an ML model to be able to classify different game stages, e.g., dialog with a non-player character, exploration, combat with enemy, in-game menu etc. In order to do that, I am ...
1
vote
0answers
29 views

Multi-armed bandits: reducing stochastic multi-armed bandits to bernoulli bandits

Agrawal and Goyal (http://proceedings.mlr.press/v23/agrawal12/agrawal12.pdf page 3) discussed how we can extend Thompson sampling for bernoulli bandits to Thompson sampling for stochastic bandits in ...
1
vote
0answers
27 views

REINFORCE Agent suddenly drops. How to verify if it's due to catastrophic forgetting?

I am using the default implementations of REINFORCE, DQN and c51 available from the tf.agents repo (links). As you can see, DQN manages to improve performance while REINFORCE seems to suffer from ...
1
vote
0answers
29 views

Should the exploration rate be updated at the end of the episode or at every step?

My agent uses an $\epsilon$-greedy strategy to learn. The exploration rate (i.e. $\epsilon$) decays throughout the training. I've seen examples where people update $\epsilon$ every time an action is ...
1
vote
0answers
49 views

How does one stack multiple observations in the input layer of a convolutional neural network?

The paper, Deep Recurrent Q-Learning for Partially Observable MDPs, talks about stacking multiple observations in the input of a convolutional neural network. How does this exactly work? Do the ...
1
vote
0answers
31 views

What is the difference between step_model and train_model in the OpenAI implementation of the A2C algorithm?

I'm struggling a little with understanding the OpenAI implementation of A2C in the baselines (version 2.9.0) package. From my understanding, one ...
1
vote
0answers
69 views

How can I evaluate a reinforcement learning algorithm over an entire problem space?

I am working on implementing an RL agent and I want to demonstrate its effectiveness over a bounded problem space. The setting is essentially a queueing network and so it can be represented as a graph....
1
vote
0answers
43 views

Which reinforcement learning approach to use when there are 2 collaborative agents?

Suppose we are training an environment with 2 collaborative agents with Reinforcement Learning. We define the following example: There is a midfielder and a striker. The midfielder's reward depends on ...
1
vote
0answers
35 views

Dynamically adapting activation function

I am training a network through reinforcement learning. The policy network learns rotations, but depending on the actual input (state), the output of the network should be restricted to be in certain ...
1
vote
0answers
94 views

SeqGAN - Policy gradient objective function interpretation

Could someone clear my doubt on the loss function used in SeqGAN paper . The paper uses policy gradient method to train the generator which is a recurrent neural network here. Have I interpreted the ...
1
vote
1answer
83 views

How to create a Q-Learning agent when we have a matrix as an action space?

I have a 2-dimentional matrix as an action space, the rows being a resource to be allocated, and the columns are the users that we will allocate the resources to. (I built my own RL environment) The ...
1
vote
0answers
42 views

What are the disadvantages of actor-only methods with respect to value-based ones?

While the advantages of actor-only algorithms, the ones that search directly the policy without the use of the value function, are clear (possibility of having a continuous action space, a stochastic ...
1
vote
0answers
114 views

To solve chess with deep RL and MCTS, how should I represent the input (the state) to a neural network?

I'm wanting to build a NN that can create a policy for each possible state. I want to combine this with MCTS to eliminate randomness so when expansion occurs, I can get the probability of the move to ...
1
vote
0answers
12 views

Relation between a value function of an MDP and a value function of the corresponding latent MDP

In paper "DeepMDP: Learning Continuous Latent Space Models for Representation Learning", Gelada et al. state in the beginning of section 2.4 The degree to which a value function of $\bar{\...
1
vote
1answer
40 views

What framework for a project with a custom environment?

I'm planning an RL project and I have to decide which RL framework do I use if any at all. The project has a highly custom environment, and testing different algorithms will be required to obtain ...
1
vote
1answer
50 views

What are some suitable positive functions as activations of neural networks?

I am working on a deep Q-learning project. My project is different than normal deep Q-learning. The rewards of my neural network must be positive because I need their values to importance sample ...
1
vote
0answers
80 views

Use of virtual worlds (e.g. Second Life) for training Artificial General Intelligence agents?

There is emerging effort for Third Wave Artificial Intelligence (Artificial General Intelligence) (http://hlc.doc.ic.ac.uk/3AI_HLC_2019.html and https://www.darpa.mil/work-with-us/ai-next-campaign) ...
1
vote
0answers
51 views

Why would DDPG with Hindsight Experience Replay not converge?

I am trying to train a DDPG agent augmented with Hindsight Experience Replay (HER) to solve the KukaGymEnv environment. The actor and critic are simple neural networks with two hidden layers (as in ...
1
vote
0answers
31 views

How to understand this NN architecture?

I was reading a paper Multi-Agent Reinforcement Learning for Adaptive User Association in Dynamic mmWave Networks and I was stuck understanding the deep neural network architecture that was used. The ...
1
vote
0answers
15 views

is it ok to take random actions while training a3c as in below code

i am trying to train an A3C algorithm but I am getting same output in the multinomial function. can I train the A3C with random actions as in below code. can someone expert comment. ...
1
vote
0answers
49 views

Why scaling reward drastically affects performance?

I have devised an gridworld-like environment where a RL agent is tasked to cover all the blank squares by passing through them. Possible actions are up, down, left, right. The reward scheme is the ...
1
vote
1answer
107 views

How is exponential moving average computed in deep Q networks?

In normal Q-learning, the update rule is an implementation of the exponential moving average, which then converges to the optimal true Q values. However, looking at DQN, how exactly is the exponential ...
1
vote
0answers
30 views

Customized food for persons based on their profile using Reinforcement learning

I am newbie to Reinforcement Learning, this is my idea - Agent(food provider) has to select a food based on the environment(based on the user profile). Here the reward will be given to the agent based ...
1
vote
0answers
33 views

How can I derive n-step off-policy temporal difference formula?

I was reading the book "Reinforcement Learning: An Introduction" by Sutton and Barto. In section 7.3, they write the formula for n-step off-policy TD as $$V(S_t) = V(S_{t-1}) + \alpha \rho_{...
1
vote
0answers
47 views

How does DQN convergence work in reinforcement learning

In supervised learning we have an unbiased target value, but in reinforcement learning this isn’t the case The network predicts its own target value, now how exactly does it converge if the network ...
1
vote
0answers
25 views

When past states contain useful information, does A3C perform better than TD3, given that TD3 does not use an LSTM?

I am trying to build an AI that needs to have some information about the past states as well. Therefore, LSTMs are suitable for this. Now, I want to know that for a problem/game like Breakout, where ...
1
vote
0answers
32 views

How can deep Q-learning converge if the targets may not be correct?

In deep Q-learning, $Q(s, a)$ and $Q'(s, a)$ are predicted or estimated by the neural network itself. In supervised learning, the target value is a true unbiased value. However, this isn't the case in ...
1
vote
0answers
43 views

What does self-play in reinforcement learning lead to?

Suppose, instead of playing against a random opponent, the reinforcement learning algorithm described above played against itself, with both sides learning. What do you think would happen in this case?...
1
vote
0answers
71 views

What happens if our target network overestimates the value?

When we use DDQN, we often use the target network in case our online network overestimates a value, but this doesn't make sense to me, because What happens if our target network is the one that ...
1
vote
0answers
57 views

What is the best way to make a deep reinforcement learning environment with a continuous 2D action space?

I understand that the actor-critic method is probably where I want to start because of how it works with continuous action spaces. However, the problem I am trying to solve would require the action be ...

1
4 5
6
7 8
13