Questions tagged [reinforcement-learning]

For questions related to learning controlled by external positive reinforcement or negative feedback signal or both, where learning and use of what has been thus far learned occur concurrently.

Filter by
Sorted by
Tagged with
0
votes
1answer
8 views

Reinforcement learning vs. Genetic Algorithms [Applications]

I'm looking for a breakdown of areas of applications for Reinforcement Learning & Neural Networks vs. Genetic Algorithms, both actual and theoretical. Links are welcome, but please provide some ...
4
votes
2answers
4k views

Negative reward (penalty) in policy gradient reinforcement learning

I am using policy gradients in my reinforcement learning algorithm, and occasionally my environment provides a severe penalty when a wrong move is made. I'm using a neural network with stochastic ...
2
votes
1answer
20 views

Doubt in Deep-Q learning with sparse rewards

I am working on a deep reinforcement learning problem, when I got stuck at the following questions. They are rather general and not specific to my specific problem. The solution uses a sparse reward ...
2
votes
1answer
25 views

Q Learning for FrozenLake environment not converging to V* values from Value Iteration

I am trying to learn tabular Q learning, value iteration using the classical algorithms (no neural networks) by using a table of states and actions. I was trying it out on FrozenLake environment in ...
2
votes
1answer
29 views

What are sim2sim, sim2real and real2real?

Recently, I always hear about the terms sim2sim, sim2real and real2real. Will anyone explain the meaning/motivation of these terms (in DL/RL research community)? What are the challenges in this ...
1
vote
3answers
132 views

How can I develop a prediction algorithm for a game of chance?

How can I develop a prediction algorithm in the case of games of chance? Suppose there is a 50:50 chance of winning. Is there way of creating a prediction algorithm?
1
vote
2answers
62 views

Running 2 NEAT nets on the same observations

So i have been playing around with neat-python. I made a program, applying neat, to play pinball on the Atari 2600. The code for that can be found in the file ...
3
votes
2answers
65 views

What is the difference between return and expected return?

At a time step $t$, for a state $S_{t}$, the return is defined as the discounted cumulative reward from that time step $t$. If an agent is following a policy (which in itself is a probability ...
0
votes
0answers
16 views

Ideas on how to train an AI to play Mario Kart with the DeSmuME Emulator

Gday guys, i have this idea in my mind for quite a while. I want to teach an AI to play Mario Kart on the NDS, which can hopefully beat me and my friends one day. Iam familiar with the theoretical ...
2
votes
0answers
25 views

Best approach for online Machine Translation with few hundred of samples?

I want to implement a model that improves itself with the passage of time. My main task is to build a machine translator (from English to Urdu).. The problem I am facing is that I have very little ...
0
votes
1answer
50 views

Model-based Reinforcement Learning algorithm for real-time robotics task

I'm quite a newbie when it comes to practically working with Deep Learning techniques, although I studied them quite a lot theoretically in the last months. However, now I'm facing my first practical ...
3
votes
0answers
40 views

Sampling in TRPO or PPO

In the TRPO paper, the objective to maximize is (equation 14) $$ \mathbb{E}_{s\sim\rho_{\theta_\text{old}},a\sim q}\left[\frac{\pi_\theta(a|s)}{q(a|s)} Q_{\theta_\text{old}}(s,a) \right] $$ which ...
5
votes
2answers
96 views

Why don't people use projected Bellman error with deep neural networks?

Projected Bellman error has shown to be stable with linear function approximation. The technique is not at all new. I can only wonder why this technique is not adopted to use with non-linear function ...
2
votes
1answer
40 views

How can I teach a computer to play N64 games using Neural Nets?

I would like to work on a project where I teach an NN to play N64 games. To my current understanding, I would need an emulator? I can do the Machine Learning side of it, im just unsure how I can ...
2
votes
0answers
22 views

Is there a way to do reinforcement learning in POMDP?

Are there any algorithms to use reinforcement learning to learn optimal policies in partially observable Markov decision process (POMDP) i.e. when the state is not perfectly observed. More ...
5
votes
1answer
123 views

What is a weighted average in a non-stationary k-armed bandit problem?

In the book Reinforcement Learning: An Introduction (page 25), by Richard S. Sutton and Andrew G. Barto, there is a discussion of the k-armed bandit problem, where the expected reward from the bandits ...
0
votes
0answers
12 views

Choosing best combinations from all possible combination expressions based few variables, unary operators, binary operators

I have a few financial variables of a stock universe like OHLC prices, volume, and other fundamentals with varying time-frequency. Using this set I'm creating an expression that gives the weights to ...
2
votes
2answers
62 views

How do we get the true value in the prediction objective in reinforcement learning?

In the book "Reinforcement Learning: An Introduction" (2018) Sutton and Barto define the prediction objective ($\overline{VE}$) as follows (page 199): $$\overline{VE}\doteq\sum_{s\epsilon S} \mu(s)[v_{...
1
vote
1answer
29 views

Can reinforcement learning be utilized for creating a simulation?

According to the definition, the AI agent has to play a game by it's own. A typical domain is the blocksworld problem. The AI determines which action the robot in a game should execute and a possible ...
1
vote
1answer
39 views

What is the difference between the definition of a stationary policy in reinforcement learning and contextual bandit?

A stationary policy is a function that maps a state to a probability distribution of actions. In a contextual bandit problem, a state itself does not include the history. But in a reinforcement ...
2
votes
1answer
50 views

Markov property in maze solving problem in reinforcement learning

By definition, every state in RL has Markov property, which means that the future state depends only on the current state, not the past states. However, I saw that in some case we can define a state ...
4
votes
1answer
41 views

How would one implement a multi-agent environment with asynchronous action and rewards per agent?

In a single agent environment, the agent takes an action, then observes the next state and reward: ...
1
vote
1answer
33 views

Unable to replicate Figure 2.1 from “Reinforcement Learning: An Introduction”

The author explains in 2.2 Action-Value Methods: To roughly assess the relative effectiveness of the greedy and $\varepsilon $-greedy methods, we compared them numerically on a suite of test ...
3
votes
1answer
30 views

Why does having a fixed policy change a Markov Decision Process (MDP) to a Markov Reward Process (MRP)?

If a policy is fixed, it is said that an MDP becomes an MRP. Why is this so? Aren't the transitions and rewards still parameterized by the action and current state? In other words, aren't the ...
4
votes
2answers
55 views

Why is having low variance important in offline policy evaluation of reinforcement learning?

Intuitively, I understand that having an unbiased estimate of a policy is important because being biased just means that our estimate is distant from the truth value. However, I don't understand ...
2
votes
1answer
99 views

What is the internal state of a Simple Neural Attentive Meta-Learner(SNAIL)?

In the paper A Simple Neural Attentive Meta-Learner, the authors mentioned right before Section 3.1: we preserve the internal state of a SNAIL across episode boundaries, which allows it to have ...
3
votes
3answers
395 views

How to implement a Continuous Control of a quadruped robot with Deep Reinforcement Learning in Pybullet and OpenAI Gym?

Description I have designed this robot in URDF format and its environment in pybullet. Each leg has a minimum and maximum value of movement. What reinforcement algorithm will be best to create a ...
2
votes
0answers
25 views

how to use Softmax action selection algorithm in atari-like game

I'm currently writing a program using keras (python 3) to play a game similar to Atari games, only in this one there are objects moving in the screen in different angles and directions (in most of ...
1
vote
0answers
23 views

RF: How to deal with environments changing state due to external factors

I have a use case where the state of the environment could change due to random events in between time steps that the agent takes actions. For example, at t1, the agent takes action a1 and is given ...
1
vote
0answers
12 views

TD losses are descreasing, but also rewards are decreasing, increasing sigma

I'm using Q-learning with some extensions such as noisy linear layers, n-steps and double DQN. The training, however, isn't that successful, my rewards are descreasing over time after a steep ...
5
votes
1answer
33 views

Is the agent aware of a possible different set of actions for each state?

I have a use case where the set of actions is different for different states. Is the agent aware of what actions are valid for which states or is the agent only aware of the entire action space (in ...
0
votes
2answers
41 views

How should I interpret the weights file of the Leela Zero neural network?

I am trying to understand the NN architecture given at https://github.com/leela-zero/leela-zero/blob/next/training/caffe/zero.prototxt. So, I downloaded the NN weights from http://zero.sjeng.org/. ...
1
vote
1answer
72 views

A3C fails to solve MountainCar-v0 enviroment (implementation by OpenAi gym)

While I've been able to solve MountainCar-v0 using Deep Q learning, no matter what I try I can't solve this enviroment using policy-gradient approaches. As far as I learnt searching the web, this is a ...
2
votes
1answer
58 views

Is it a good idea to store the policy in a database?

I'm a beginner in ML and have been researching RL quite a bit recently. I'm planning to create an RL application to play a zero-sum game. This will be web-based, so anyone can play it. I wondered if ...
0
votes
1answer
32 views

Why doesn't stability in prediction imply stability in control in off-policy reinforcement learning?

Prediction's goal is to get an estimate of a performance of a policy given a specific state. Control's goal is to improve the policy wrt. the prediction. The alternation between the two is the ...
2
votes
1answer
131 views

Large and Multiple-actions space

I have a steady hex-map and turn-based wargame featuring WWII carrier battles On a given turn, a player may choose different and independent actions 
(moving one, two naval unit, assigning a mission ...
5
votes
2answers
385 views

Reinforcement Learning with asynchronous feedback

I want suggestions on literature on Reinforcement Learning algorithms that perform well with asynchronous feedback from the environment. What I mean by asynchronous ...
0
votes
0answers
16 views

DQN unlearns certain OpenAI-Gym environments

I solved the OpenAI-Gym MountainCar-v0 environment using dqn(using low-state-dimensional input). When I used the same code for solving CartPole-v0 environment, the network got trained in the reverse ...
8
votes
5answers
322 views

What's a good resource for getting familiar with reinforcement learning?

I am familiar with supervised and unsupervised learning. I did the SaaS course done by Andrew Ng on Coursera.org. I am looking for something similar for reinforcement learning. Can you recommend ...
1
vote
1answer
56 views

Where are reinforcement algorithms used in financial services?

One of the most common misconceptions about reinforcement learning (RL) applications is that, once you deploy them, they continue to learn. And, usually, I'm left having to explain this. As part of my ...
3
votes
0answers
34 views

What is the difference between random and sequential sampling from the reply memory?

I was working on an RL problem and I am confused at one specific point. We use replay memory so that the network learns about previous actions and how these actions lead to a success or a failure. ...
3
votes
1answer
38 views

What are the differences between stability and convergence in reinforcement learning?

The terms are mentioned in the paper: “An Emphatic Approach to the Problem of off-Policy Temporal-Difference Learning.” (Sutton, Mahmood, White; 2016) and more, of course. In which paper, they ...
2
votes
1answer
45 views

What happens to the optimal value function if the reward is multiplied by a constant?

What happens to the optimal action-value function, $q_*$ if the reward is multiplied by a constant $c$? Is the optimal action-value function also multiplied by such a constant?
1
vote
1answer
97 views

How does Hindsight Experience Replay learn from unsuccessful trajectories

I am confused by how HER learns from unsuccessful trajectories. I understand that from failed trajectories it creates 'fake' goals that it can learn from. Ignoring HER for now, if in the case where ...
0
votes
1answer
46 views

LSTM in reinforcement learning

Please tell me that is the LSTM network for the problem of reinforcement learning, as I explain to her what she will get the reward of a prediction, because the output will contain only actions? Well,...
-1
votes
1answer
25 views

Does higher Accuracy in Reinforcement Learning indicate better model performance?

If a reinforcement learning algorithm uses a Deep Neural Network to predict the action given a state (a NN for a policy function), an Monte Carlo Tree Search in a model-based learning setup, then ...
2
votes
1answer
62 views

Monte Carlo learning for Reinforcement learning

When you train a model using Monte Carlo-based learning the state and action taken at each step is recorded, and then at some point an end state is reached and the agent receives some reward - what do ...
0
votes
0answers
17 views

A2C for the game of Hanabi underfits

I am trying to solve the game of Hanabi (paper describing game) with actor-critic algorithm. I took code for the environment from the Deepmind's repository and implemented a2c algorithm myself. From ...
3
votes
1answer
31 views

Reinforcement learning with hints or reference model

In Reinforcement Learning, when I train a model, it comes up with its own set of solutions. For example, if I am training a robot to walk, it will come up with its own walking gait, such as this Deep ...
2
votes
1answer
38 views

How do policy gradients compute an infinite probability distribution from a neural network

Do neural networks compute the probability distribution for policy gradient methods. If so, how do they compute an infinite probability distribution? How do you represent a continuous action policy ...