Questions tagged [reinforcement-learning]

For questions related to learning controlled by external positive reinforcement or negative feedback signal or both, where learning and use of what has been thus far learned occur concurrently.

Filter by
Sorted by
Tagged with
2
votes
0answers
64 views

Has anyone been able to solve OpenAI's hardcore bipedal walker with their implementation of DDPG?

As the question suggests, I'm trying to see if I can solve OpenAI's hardcore version of their gym's bipedal walker using OpenAI's DDPG algorithm. Below is a performance graph from my latest attempt, ...
1
vote
0answers
46 views

Is it possible to use Reward Function of type R(s, a, s') if more than one action is applied?

I am applying a reinforcement learning agent (PPO2, stable baselines implementation) to a custom built environment using OpenAI Gym. One reward function (formualted as loss function, that is, all ...
1
vote
0answers
30 views

How is computed the gradient with respect to each output node from a loss value?

newbie here. I am studying the REINFORCE method in "Deep Reinforcement Learning Hands-On". I can't understand how, after computing the loss of the episode, that loss is backpropagated in a NN with ...
1
vote
1answer
75 views

Deep Q Learning Algorithm for Simple Python Game makes player stuck

I made a simple Python game. A screenshot is below: Basically, a paddle moves left and right catching particles. Some make you lose points while others make you gains points. This is my first Deep Q ...
1
vote
0answers
26 views

What is a high performing network architecture to use in a PPO2 MlpLnLstmPolicy RL model?

I am playing around with creating custom architectures in stable-baselines. Specifically I am training an agent using a PPO2 model. My question is, are there some rules of thumb or best practices in ...
2
votes
0answers
19 views

Why do these reward functions give different training curves?

Let's say our task is to pick and place a block, like: https://gym.openai.com/envs/FetchPickAndPlace-v0/ Reward function 1: -1 for block not placed, 0 for block placed Reward function 2: 0 for block ...
2
votes
1answer
59 views

Is it a good idea to store the policy in a database?

I'm a beginner in ML and have been researching RL quite a bit recently. I'm planning to create an RL application to play a zero-sum game. This will be web-based, so anyone can play it. I wondered if ...
2
votes
1answer
33 views

How are the observations stored in the RNN that encodes the state?

I am a bit confused about observations in RL systems which use RNN to encode the state. I read a few papers like this and this. If I were to use a sequence of raw observations (or features) as an ...
1
vote
0answers
28 views

How to solve optimal control problem with reinforcement learning

The problem I am trying to attack is a predator-prey pursuit problem. There are multiple predators that pursue multiple preys and preys tried to evade predators. I am trying to solve a simplified ...
1
vote
0answers
24 views

Can reinforcement learning be used to map a terrain?

Recently, I was training an agent on a Windy Gridworld terrain with stochastic winds. From the agent's behavior, it was clear that the agent was trying to avoid places with large winds that'll throw ...
1
vote
0answers
61 views

How to use the LSTM layer in PPO architecture?

What is the best way of using the LSTM layer in PPO architecture? Should I use them in the first layer of both actor and critic, or use them just before the final layer of these networks? Should I ...
3
votes
1answer
97 views

How do I solve this optimal control problem with reinforcement learning?

I am new to reinforcement learning. I would like to solve an optimal control problem with reinforcement learning. The objective is for a wolf to catch a rabbit. The wolf and the rabbit run on a ...
1
vote
1answer
83 views

How is Monte Carlo different from model-based methods?

I was going through an article where it is mentioned: The Monte-Carlo methods require only knowledge base (history/past experiences)—sample sequences of (states, actions and rewards) from the ...
2
votes
0answers
40 views

Coloring graphs with reinforcement learning

I am trying to build an RL agent to solve the NP-hard problem graph coloring. The problem is quite challenging. This how I addressed it. The environment To preserve the scalability of the ...
2
votes
0answers
48 views

What is the meaning of the words 'bias' and 'variance' in RL?

In algorithms like MC/TD (tabular value approximation) two of the metrics used to measure their performance are the bias and the variance. What do these terms mean? And which characteristic of the ...
0
votes
0answers
13 views

Derivation for Value Iteration of CVaR

I am reading a paper named Risk-sensitive and Robust Decision-making: a CVaR Optimization Approach. In appendix A.3 they provide a proof for their Theorem $4$. The $n=1$ case for equation (11) is ...
0
votes
1answer
41 views

Entropy term in Proximal Policy Optimization (PPO) becomes undefined after few training epochs

I have implemented the total loss of my PPO objective as follows:- ...
0
votes
1answer
125 views

OpenAI Spinning Up: Breakout-v0 example

I haven't been able to find any assistance / examples which could help me implement OpenAI's Spinning Up resource to solve Atari's Breakout-v0 game in the OpenAI gym. I simply want to know why the ...
0
votes
1answer
35 views

Why is On-Policy MC/TD Algorithm guaranteed to converge to optimal policy?

Let's say we have a task where the cost depends entirely on the path length to a terminal state, so the goal of an agent would be to take actions to reach terminal state as quickly as possible. Now ...
1
vote
0answers
37 views

Reward problem in A2C with multiple simultaneous discrete actions

I've built an A2C model whose actor's network has two different kinds of discrete actions, so the critic would take state and action (note that critic takes 2 actions because in each timestep we will ...
1
vote
0answers
37 views

Unique game problem (ML, DP, PP etc)

Looking for a solution to my below game problem. I believe it to require some sort of reinforcement learning, dynamic programming, or probabilistic programming solution, but am unsure... This is my ...
2
votes
0answers
95 views

What is ratio of the objective function in the case of continuous action spaces?

I'm trying to implement the proximal policy optimization (PPO) algorithm. I'm confused on how to make it work with continuous action space. For discrete action space, the output of the network is the ...
2
votes
1answer
34 views

Will the target network, which is less trained than the normal network, output inferior estimates?

I'm having some trouble understanding some parts of the usage of target networks. I get that having the same network predict the state/action/advantage values for both the current networks can lead ...
3
votes
0answers
69 views

How can I use Q-learning for inventory decision making?

I am trying to model operational decisions in inventory control. The control policy is base stock with a fixed stock level $S$. That is replenishment orders are placed for every demand arrival to take ...
2
votes
0answers
28 views

Will the RL agent implemented as a neural network fine-tune itself?

Normally, when you develop a neural network, train it for object recognition (on normal objects like bike, car, plane, dog, cloud, etc.), and it turns out to perform very well, you would like to fine-...
0
votes
1answer
48 views

Probabilistic action selection in pursuit algorithm

In the Pursuit algorithm (to balance exploration and exploitation), the greedy action has a probability say $p_1$ (updated every episode) of being selected, while the rest have a probability $p_2$ (...
3
votes
1answer
56 views

How does RL based neural architecture search work?

I have read through many of the papers and articles linked in this thread but I haven't been able to find an answer to my question. I have built some small RL networks and I understand how REINFORCE ...
11
votes
1answer
235 views

How to stay a up-to-date researcher in ML/RL community?

As a student who wants to work on machine learning, I would like to know how it is possible to start my studies and how to follow it to stay up-to-date. For example, I am willing to work on RL and MAB ...
2
votes
1answer
25 views

Using heuristic dense rewards in a sparse problem

If I am training an agent to try and navigate a maze as fast as possible, a simple reward would be something like \begin{align} R(\text{terminal}) &= N - \text{time}\ \ , \ \ N \gg \text{...
2
votes
1answer
80 views

Several questions related to UCT and MCTS

In Bandit Based Monte-Carlo Planning, the article where UCT is introduced as a planning algorithm, there is an algorithm description in page 285 (4 of the pdf). Comparing this implementation of UCT (...
2
votes
2answers
39 views

Can non-Markov environments also be deterministic?

The definition of deterministic environment I am familiar with goes as follows: The next state of the agent depends only on the current state and the action chosen by the agent. By exclusion, ...
3
votes
1answer
48 views

Can one agent command another agent in a multi-agent reinforcement learning setting?

In reinforcement learning, an agent is usually fully autonomous and independent. It executes actions on the environment, but no other agent can control, explore or command this agent. In multi-agent ...
1
vote
0answers
33 views

Can multiple reinforcement algorithms be applied to the same system?

Can a system, for instance robotic vehicle, be controlled by more than one reinforcement learning algorithm. I intend to use one to address collision avoidance whereas the other to tackle autonomous ...
1
vote
0answers
15 views

Can the agent of reinforcement learning system serve as the environment for other agents and expose actions as services?

Can the agent of reinforcement learning system serve as the environemnt for other agents and expose actions as services? Are there research that consider such question? I tried to formulate the ...
1
vote
2answers
43 views

How to enforce covariance-matrix output as part of the last layer of a Policy Network?

I have a continuous state space, and a continuous action space. The way I understand it, I can build a policy network which takes as input a continuous state vector and outputs both mean vector and ...
2
votes
0answers
63 views

How does Friend-or-Foe Q-learning intuitively work?

I read about Q-Learning and was reading about multi-agent environments. I tried to read the paper Friend-or-Foe Q-learning, but could not understand anything, except for a very vague idea. What does ...
1
vote
0answers
61 views

How do I determine the generalisation ability of a neural network?

I am trying to ascertain if my neural network is able to generalize or if it’s simply using memory/overfitting to solve a task. I would like my model to generalise. Currently, I train the neural ...
1
vote
1answer
40 views

Understanding TD(0) algorithm implementation

I came across the $TD(0)$ algorithm from Sutton and Barto: Clearly, the only difference of TD methods with the MC methods is that TD method is not waiting till the end of the episode to update the $...
0
votes
0answers
26 views

Why epsilon-greedy hyperparameter is annealed smoothly?

Regarding of DQN, or DQRNN, (reinforcement learning) To me, RL is a process that can be divided into 2 stages: Exploring wide range of paths (acting randomly) Refining the current optimal paths (...
1
vote
1answer
54 views

Why does GLIE+MC Control Algorithm use a single episode of Monte Carlo evaluation?

GLIE+MC control Algorithm: My question is why does this algorithm use only a single Monte Carlo episode (during PE step) to compute the $Q(s,a)$? In my understanding this has the following drawbacks: ...
0
votes
0answers
34 views

What is a non-stationary mean in the context of RL? [duplicate]

I have come across the expression non-stationary mean in the RL lectures by David Silver, and I really could not understand this expression and its difference from a normal mean. So what exactly is ...
1
vote
0answers
57 views

How can I develop this ML/AI system that I want to use in my new mobile app?

I have an idea for a new mobile app. Here is what I want to accomplish using AI; I want to get an image (png format), (maybe just byte data too), from my application (I'm developing with Unity3D/C#), ...
0
votes
1answer
64 views

What are options in reinforcement learning?

According to a lecture about Reinforcement Learning, the concept of options allows searching the state space of an agent much faster. The lecture came from Nptel [1] (National Program on Technology ...
1
vote
2answers
170 views

Number of states in taxi environment (Dietterich 2000)

Dietterich, who introduced the taxi environment (see p. 9), states the following: In total there “are 500 [distinct] possible states: 25 squares, 5 locations for the passenger (counting the four ...
1
vote
1answer
54 views

Importance of initialisation of State-Action/State values in RL

I was wondering is there any empirical/theoretical evidence on the effect of initial values of State-Action/State values on the training of an RL agent (the values an RL agent assigns to visited ...
0
votes
0answers
31 views

Questions performance SimPLe pong for AI demo

For a demo I need to develop an AI solution to learn how to play pong. I have the following requirements: Computer needs to play against a human player. Learn while playing the game. Poor AI result ...
1
vote
0answers
49 views

Doubt regarding improvement of State Value by n-step returns

Excerpt from Sutton and Barto: The expected value of all $n$-step returns is guaranteed to improve in a certain way over the current value function as an approximation to the true value ...
0
votes
1answer
75 views

Is DDPG just for deterministic environments?

I want to develop an AI for continuous space. I reached to DDPG algorithm that takes actions deterministically. If DDPG takes actions deterministically, should the environment also be deterministic? ...
6
votes
1answer
158 views

Does AlphaZero use Q-Learning?

I was reading the AlphaZero paper Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm, and it seems they don't mention Q-Learning anywhere. So does AZ use Q-...
3
votes
1answer
64 views

DQN in stochastic environment

I'm trying to apply a DQN to a stochastic environment but i'm having trouble getting it to converge. I found some similar questions asked here, but no solutions yet. I can fairly easy get the DQN to ...