Questions tagged [reinforcement-learning]
For questions related to learning controlled by external positive reinforcement or negative feedback signal or both, where learning and use of what has been thus far learned occur concurrently.
667
questions
0
votes
0answers
5 views
What are sim2sim, sim2real, real2real; and challenges?
Recently, I always hear about the terms like sim2sim, sim2real, real2real. Will anyone explain the meaning/motivation of these terms (in DL/RL research community)?
What are challenges in these ...
0
votes
0answers
13 views
Ideas on how to train an AI to play Mario Kart with the DeSmuME Emulator
Gday guys,
i have this idea in my mind for quite a while. I want to teach an AI to play Mario Kart on the NDS, which can hopefully beat me and my friends one day. Iam familiar with the theoretical ...
2
votes
0answers
23 views
Best approach for online Machine Translation with few hundred of samples?
I want to implement a model that improves itself with the passage of time.
My main task is to build a machine translator (from English to Urdu).. The problem I am facing is that I have very little ...
0
votes
1answer
48 views
Model-based Reinforcement Learning algorithm for real-time robotics task
I'm quite a newbie when it comes to practically working with Deep Learning techniques, although I studied them quite a lot theoretically in the last months. However, now I'm facing my first practical ...
2
votes
1answer
40 views
How can I teach a computer to play N64 games using Neural Nets?
I would like to work on a project where I teach an NN to play N64 games.
To my current understanding, I would need an emulator?
I can do the Machine Learning side of it, im just unsure how I can ...
3
votes
0answers
36 views
Sampling in TRPO or PPO
In the TRPO paper, the objective to maximize is (equation 14)
$$
\mathbb{E}_{s\sim\rho_{\theta_\text{old}},a\sim q}\left[\frac{\pi_\theta(a|s)}{q(a|s)} Q_{\theta_\text{old}}(s,a) \right]
$$
which ...
2
votes
0answers
21 views
Is there a way to do reinforcement learning in POMDP?
Are there any algorithms to use reinforcement learning to learn optimal policies in partially observable Markov decision process (POMDP) i.e. when the state is not perfectly observed. More ...
0
votes
0answers
12 views
Choosing best combinations from all possible combination expressions based few variables, unary operators, binary operators
I have a few financial variables of a stock universe like OHLC prices, volume, and other fundamentals with varying time-frequency. Using this set I'm creating an expression that gives the weights to ...
1
vote
1answer
29 views
Can reinforcement learning be utilized for creating a simulation?
According to the definition, the AI agent has to play a game by it's own. A typical domain is the blocksworld problem. The AI determines which action the robot in a game should execute and a possible ...
1
vote
1answer
37 views
What is the difference between the definition of a stationary policy in reinforcement learning and contextual bandit?
A stationary policy is a function that maps a state to a probability distribution of actions.
In a contextual bandit problem, a state itself does not include the history. But in a reinforcement ...
2
votes
1answer
49 views
Markov property in maze solving problem in reinforcement learning
By definition, every state in RL has Markov property, which means that the future state depends only on the current state, not the past states.
However, I saw that in some case we can define a state ...
4
votes
1answer
40 views
How would one implement a multi-agent environment with asynchronous action and rewards per agent?
In a single agent environment, the agent takes an action, then observes the next state and reward:
...
1
vote
1answer
33 views
Unable to replicate Figure 2.1 from “Reinforcement Learning: An Introduction”
The author explains in 2.2 Action-Value Methods:
To roughly assess the relative effectiveness of the greedy and $\varepsilon $-greedy methods, we compared them numerically on a suite of test ...
3
votes
1answer
29 views
Why does having a fixed policy change a Markov Decision Process (MDP) to a Markov Reward Process (MRP)?
If a policy is fixed, it is said that an MDP becomes an MRP.
Why is this so? Aren't the transitions and rewards still parameterized by the action and current state? In other words, aren't the ...
4
votes
2answers
55 views
Why is having low variance important in offline policy evaluation of reinforcement learning?
Intuitively, I understand that having an unbiased estimate of a policy is important because being biased just means that our estimate is distant from the truth value.
However, I don't understand ...
2
votes
0answers
25 views
how to use Softmax action selection algorithm in atari-like game
I'm currently writing a program using keras (python 3) to play a game similar to Atari games, only in this one there are objects moving in the screen in different angles and directions (in most of ...
1
vote
0answers
23 views
RF: How to deal with environments changing state due to external factors
I have a use case where the state of the environment could change due to random events in between time steps that the agent takes actions. For example, at t1, the agent takes action a1 and is given ...
1
vote
0answers
12 views
TD losses are descreasing, but also rewards are decreasing, increasing sigma
I'm using Q-learning with some extensions such as noisy linear layers, n-steps and double DQN.
The training, however, isn't that successful, my rewards are descreasing over time after a steep ...
5
votes
1answer
33 views
Is the agent aware of a possible different set of actions for each state?
I have a use case where the set of actions is different for different states. Is the agent aware of what actions are valid for which states or is the agent only aware of the entire action space (in ...
0
votes
0answers
16 views
DQN unlearns certain OpenAI-Gym environments
I solved the OpenAI-Gym MountainCar-v0 environment using dqn(using low-state-dimensional input). When I used the same code for solving CartPole-v0 environment, the network got trained in the reverse ...
3
votes
0answers
34 views
What is the difference between random and sequential sampling from the reply memory?
I was working on an RL problem and I am confused at one specific point. We use replay memory so that the network learns about previous actions and how these actions lead to a success or a failure.
...
0
votes
1answer
32 views
Why doesn't stability in prediction imply stability in control in off-policy reinforcement learning?
Prediction's goal is to get an estimate of a performance of a policy given a specific state.
Control's goal is to improve the policy wrt. the prediction.
The alternation between the two is the ...
3
votes
1answer
38 views
What are the differences between stability and convergence in reinforcement learning?
The terms are mentioned in the paper: “An Emphatic Approach to the Problem of off-Policy Temporal-Difference Learning.” (Sutton, Mahmood, White; 2016) and more, of course.
In which paper, they ...
-1
votes
1answer
25 views
Does higher Accuracy in Reinforcement Learning indicate better model performance?
If a reinforcement learning algorithm uses a Deep Neural Network to predict the action given a state (a NN for a policy function), an Monte Carlo Tree Search in a model-based learning setup, then ...
0
votes
0answers
17 views
A2C for the game of Hanabi underfits
I am trying to solve the game of Hanabi (paper describing game) with actor-critic algorithm. I took code for the environment from the Deepmind's repository and implemented a2c algorithm myself. From ...
2
votes
1answer
38 views
How do policy gradients compute an infinite probability distribution from a neural network
Do neural networks compute the probability distribution for policy gradient methods. If so, how do they compute an infinite probability distribution? How do you represent a continuous action policy ...
2
votes
1answer
45 views
What happens to the optimal value function if the reward is multiplied by a constant?
What happens to the optimal action-value function, $q_*$ if the reward is multiplied by a constant $c$? Is the optimal action-value function also multiplied by such a constant?
2
votes
1answer
62 views
Monte Carlo learning for Reinforcement learning
When you train a model using Monte Carlo-based learning the state and action taken at each step is recorded, and then at some point an end state is reached and the agent receives some reward - what do ...
2
votes
1answer
47 views
Can an RL algorithm trained in one environment be successful in a different one?
Can an RL algorithm trained in one environment be successful in a different one?
For example, if I train a model to go through one labyrinth, could this model also go through a different but similar ...
0
votes
1answer
62 views
How can the derivative of a neural network be calculated, given no mathematical expression?
Neural networks (NNs) are used as approximators in reinforcement learning (RL). To update the policy in RL, the actor network's gradients w.r.t its weights are needed. Since NN doesn't have a ...
5
votes
1answer
34 views
Reinforcement learning with incremental action space
Let's say we have a problem which can be solved by some RL algorithms (DQN, for example, because we have discrete action space). At first, the action space is fixed (the number of actions is $n_1$), ...
1
vote
1answer
31 views
How is the actor-critic algorithm guaranteed to converge?
From my understanding, the critic evaluates the policy (actor) following dynamic programming (DP) or approximate dynamic programming (ADP) scheme, which should converge to the optimal value function ...
1
vote
0answers
50 views
Is Value Iteration better than Policy Iteration for first few iterations?
In Policy Iteration (PI), the action generated by the policy, whether it's optimal or not w.r.t the current value function $v(s)$. Whereas, in Value Iteration, the action is greedily generated w.r.t ...
1
vote
1answer
29 views
Can policy iteration use only the immediate reward for updates?
Is it still a policy iteration algorithm if the policy is updated optimizing a function of the immediate reward instead of the value function?
3
votes
1answer
32 views
How important is the choice of the initial state?
Is it crucial to always have the same initial (starting) state for Reinforcement Learning, for example, for Q-learning or DQN?
Or it can vary?
3
votes
1answer
54 views
What is a non-starving policy in reinforcement learning?
In the paper, Eligibility Traces for off-Policy Policy Evaluation (2010), by Doina Precup et al., mentioned the term "non-starving" many times. The specific use of the term was like "non-starving ...
3
votes
1answer
31 views
Reinforcement learning with hints or reference model
In Reinforcement Learning, when I train a model, it comes up with its own set of solutions. For example, if I am training a robot to walk, it will come up with its own walking gait, such as this Deep ...
0
votes
0answers
13 views
Invalid moves in Deep Reinforcement Learning for games [duplicate]
I've been working on a bot for a game involving dice throws and chance. The architecture involved is similar to AlphaZero in the that it has Convolutions and MCTS.
According to the current state ...
1
vote
0answers
22 views
PPO: action std or entropy for exploration?
When trying to implement my own PPO (Proximal Policy Optimizer), I came accross two different implementations :
Exploration with action std :
Collect trajectories on ...
3
votes
1answer
71 views
How is the gradient of the loss function in DQN derived?
In the original DQN paper, page 1, the loss function of the DQN is
$$
L_{i}(\theta_{i}) = \mathbb{E}_{(s,a,r,s') \sim U(D)} [(r+\gamma \max_{a'} Q(s',a',\theta_{i}^{-}) - Q(s,a;\theta_{i}))^2]
$$
...
2
votes
1answer
51 views
Can someone please help me validate my MDP?
Problem Statement :
I have a system with four states - S1 through S4 where S1 is the beginning state and S4 is the end/terminal state. The next state is always better than the previous state i.e if ...
0
votes
0answers
29 views
Finding optimal Value function and Policy for an MDP
I am solving an RL MDP problem which is model based. I have an MDP which has four possible states S1-S4 and four different actions A1-A4, with S4 being terminal state and S1 is the beginning state. ...
4
votes
1answer
66 views
Benchmarks for reinforcement learning in discrete MDPs
To compare the performance of various algorithms for perfect information games, reasonable benchmarks include reversi and m,n,k-games (generalized tic-tac-toe). For imperfect information games, ...
2
votes
1answer
42 views
Can I have different rewards for a single action based on which state it transitions to?
I am working on an MDP where there are four states and ten actions. I am supposed to derive the optimal policy to reach the desired state. At any state, a particular action can take you to any of the ...
2
votes
0answers
36 views
Feasibility of using machine learning to obtain self-consistent solutions
I am a physicist and I don't have much background on machine learning or deep learning except taking a couple of courses on statistics. In physics, we often simulate a model by means of two-way ...
2
votes
1answer
36 views
Is a state that includes only the past n-step price records partially observable?
I'm currently working on a project to make an DQN agent that decides whether to charge or discharge an electric vehicle according to hourly changing price to sell or buy. The price pattern also varies ...
3
votes
1answer
49 views
Why are model-based methods more sample efficient than model-free methods?
Why do model-based methods use fewer samples than model-free methods? Here, I'm specifically referring to model-based methods in which we have to learn a policy and model. I can only think of two ...
2
votes
1answer
31 views
Can supervised learning be recast as reinforcement learning problem?
Let's assume that there is a sequence of pairs $(x_i, y_i), (x_{i+1}, y_{i+1}), \dots$ of observations and corresponding labels. Let's also assume that the $x$ is considered as independent variable ...
1
vote
1answer
44 views
When does AlphaZero play suboptimal moves?
If AlphaZero was always playing the best moves it would just generate the same training game over and over again. So where does the randomness come from? When does it decide not to play the most ...
4
votes
2answers
139 views
What does the symbol $\mathbb E$ mean in these equations?
I came across some papers that use $\mathbb E$ in equations, in particular, this paper: https://arxiv.org/pdf/1511.06581.pdf. Here is some equations from the paper that uses it:
$Q^\pi \left(s,a \...
