Questions tagged [reinforcement-learning]
For questions related to learning controlled by external positive reinforcement or negative feedback signal or both, where learning and use of what has been thus far learned occur concurrently.
740
questions
-1
votes
0answers
29 views
Which algorithm to use?
When to use
Genetic algirithm or any reinforcement learning
When to use supervised learning
How to use which is more suitable for which?
I am a beginer.
Actually just as a test I was ...
1
vote
0answers
13 views
What's the value of making the RL agent's output stochastic opposed to deterministic?
I have a question about a reinforcement learning problem.
I'm training an agent to add or delete pixels in a [12 x 12] 2D space (going to be 3D in the future).
Its action space consists of two ...
3
votes
1answer
47 views
Why cannot an AI agent adjust the reward function directly?
In standard Reinforcement Learning the reward function is specified by an AI designer and is external to the AI agent. The agent attempts to find a behaviour that collects higher cumulative discounted ...
4
votes
1answer
267 views
Why is reinforcement learning not the answer to AGI?
I previously asked a question about How can an AI freely make decisions on a network?. I got a great answer about how current algorithms lack agency. The first thing I thought of was reinforcement ...
0
votes
0answers
24 views
Should I consider mean or sampled value for action selection in ppo algorithm?
When considering the policy network in PPO algorithm, we need to fit a Gaussian distribution to the neural network output (for a continuous action space problem). When I use this network to obtain ...
0
votes
0answers
12 views
Deciding std. deviation for policy network output?
When I try to fit a Normal Distribution to the output of a policy network, for a continuous action space problem, what should be its standard deviation? mean for the distribution will directly be the ...
0
votes
0answers
26 views
How would you differentiate between different on-policy reinforcement learning algorithms?
How would you differentiate between different on-policy reinforcement learning algorithms?
1
vote
0answers
29 views
Should an RL agent directly observe the reward?
I am training an A2C reinforcement learning agent in a dense reward environment (where rewards are known and explicit at every timestep).
Is it redundant to include the previous reward in the current ...
1
vote
0answers
37 views
Reinforcement learning for a 2D game involving two players
I'd like to create an AI for a 2D game involving two players fighting against each other. The map look something like this (The map is a NxN array somehow randomly generated):
Basically the players ...
3
votes
1answer
35 views
Does using the softmax function in Q learning not defeat the purpose of Q learning?
It is my understanding that, in Q-learning, you are trying to mimic the optimal $Q$ function $Q*$, where $Q*$ is a measure of the predicted reward received from taking action $a$ at state $s$ so that ...
1
vote
1answer
33 views
Are there OpenAI Gym continuing environments (other than inverted pendulum) and baselines?
I would like to use OpenAI Gym to solve a continuing environment, that is, a problem with a single, never-ending episode (please note I don't mean a continuous environment with continuous state and ...
2
votes
2answers
94 views
what will i be able to do in the end of AI: modern approach? [closed]
i just started the book and i was wondering , what will i be able to do in AI by the end of the book ?
and more particularly, what is my position with Reinforcement Learning, deep neural networks and ...
0
votes
1answer
42 views
Designing a reinforcement learning AI for a game of connect 4
I've made a connect 4 game in javascript, and I want to design an AI for it. I made a post the other day about what output would be needed, and I think I could use images of the board and a CNN. I did ...
2
votes
1answer
33 views
Does adding a constant to all rewards change the set of optimal policies in episodic tasks?
I'm taking a Coursera course on Reinforcement learning. There was a question there that wasn't addressed in the learning material: Does adding a constant to all rewards change the set of optimal ...
3
votes
1answer
42 views
Why is the stationary distribution independent of the initial state in the proof of the policy gradient theorem?
I was going through the proof of the policy gradient theorem here: https://lilianweng.github.io/lil-log/2018/04/08/policy-gradient-algorithms.html#svpg
In the section "Proof of Policy Gradient ...
1
vote
1answer
44 views
Building 'evaluation' neural networks for go, reversi, checkers etc, how to train?
I'm trying to build neural networks for games like Go, Reversi, Othello, Checkers, or even tic-tac-toe, not by calculating a move, but by making them evaluate a position.
The input is any board ...
2
votes
0answers
43 views
What form of output would be needed to train a model on a connect 4 AI?
I've had a big interest in machine learning for a while, and I've followed along a few tutorials, but have never made my own project. After losing many games of connect 4 with my friends, I decided to ...
0
votes
0answers
27 views
What is the difference between value iteration and policy iteration? [duplicate]
In reinforcement learning, what is the difference between policy iteration and value iteration?
As much as I understand, in value iteration, you use the Bellman equation to solve for the optimal ...
2
votes
1answer
42 views
What is the effect of picking action deterministicly at inference with Policy Gradient Methods?
In policy gradient methods such as A3C/PPO, the output from the network is probabilities for each of the actions. At training time, the action to take is sampled from the probability distribution.
...
1
vote
1answer
47 views
Where does entropy enter in Soft Actor-Critic?
I am currently trying to understand SAC (Soft Actor-Critic), and I am thinking of it as a basic actor-critic with the entropy included. However, I expected the entropy to appear in the Q-function. ...
2
votes
0answers
24 views
How would one develop an action space for a game that is proprietary?
I'm currently trying to develop an RL that will teach itself to play the popular fighting game "Tekken 7". I initially had the idea of teaching it to play generally- against actual opponents with ...
0
votes
0answers
34 views
Can supervised learning be used to solve the inverted pendulum problem?
I know that reinforcement learning has been used to solve the inverted pendulum problem.
Can supervised learning be used to solve the inverted pendulum problem?
For example, there could be an ...
0
votes
0answers
22 views
Bandits with missing contexts
Say I learn an optimal policy $\pi(a|c)$ for a contextual multi-armed bandit problem, where the context c is a composite of multiple context variables $c = c_1, c_2, c_3$. For example, the context is ...
4
votes
0answers
54 views
Unable to train Coach for Banana-v0 Gym environment
I have just started playing with Reinforcement learning and starting from the basics I'm trying to figure out how to solve Banana Gym with coach.
Essentially ...
0
votes
0answers
28 views
How can we make sure, how well the reinforcement learning works?
I read a paper which is about Deep Reinforcement Learning and it tries to use this method on stock data set. It has been showed that it reach the maximum return(profit). It has been implemented in ...
1
vote
0answers
23 views
How to set the multiple continuous actions with constraints
I want to build a Deep Reinforcement Learning Model for Asset allocation.
Background:
I have 7 stock indexes from different markets, and I want to build a policy to produce the action (likes whether ...
4
votes
0answers
44 views
Normalizing Normal Distributions in Thompson Sampling for online Reinforcement Learning
In my implementation of Thompson sampling (TS) for online Reinforcement Learning, my distribution for selecting $a$ is $\mathcal{N}(Q(s, a), \frac{1}{C(s,a)+1})$ where $C(s,a)$ is the number of times $...
2
votes
1answer
41 views
Understanding proof of lemma 1 (policy improvement bound) of the “Trust Region Policy Optimization” paper
In the Trust Region Policy Optimization paper, in Lemma 1 of Appendix A, I did not quite understand the transition from (21) from (20). In going from (20) to (21), $A^\pi(s_t, a_t)$ is substituted ...
2
votes
0answers
23 views
What could be the cause of the drop of the total reward when using DQN to solve the cart-pole environment?
I'm trying to use DQN to solve the cart-pole environment. I have 2 networks (target and behavior). Both of them have 3 hidden layers with 24 neurons, using the ReLU activation. The loss is MSE and the ...
3
votes
2answers
34 views
Effects of translating RL action probability through non linearity
I am training an RL agent (specifically using the PPO algorithm) on a game environment with 2 possible actions left or right.
The actions can be taken with varying "force"; e.g. go left 17% or go ...
3
votes
1answer
197 views
What does the Markov assumption say about the history of state sequences?
Does the Markov assumption say that the conditional probability of the next state only depends on the current state or does it say that the conditional probability depends on a fixed finite number of ...
1
vote
0answers
16 views
Finding total number of states in a POMDP
I've been working on a question that is posed in a document I've been reading, that models qualifying for a job as a POMDP. In this model, a person takes 3 exams, and must pass all of them in order to ...
2
votes
2answers
38 views
On-policy state distribution for episodic tasks on Sutton & Barto, page 199
In Sutton & Barto's "Reinforcement Learning: An Introduction", 2nd edition, page 199, they describe the on-policy distribution for episodic tasks in the following box:
I don't understand how this ...
0
votes
0answers
16 views
Same implementation, but agent is not learning in Retro Pong Environment
I tried to implement the exact same python coding by Andrej Karpathy to train RL agent to play Pong, except that I migrated the environment from Gym to Retro.
Everything is the same except the action ...
4
votes
0answers
32 views
Counterexamples to the reward hypothesis
On Sutton and Barto's RL book, the reward hypothesis is stated as
that all of what we mean by goals and purposes can be well thought of as the maximization of the expected value of the cumulative ...
4
votes
2answers
102 views
Are there any online competitions for Reinforcement Learning?
Kaggle is limited to only supervised learning problems. There used to be www.rl-competition.org but they've stopped.
Is there anything else I can do other than locally trying out different algorithms ...
2
votes
1answer
49 views
Can OpenAI simulations be used in real world applications?
I know that classical control systems have been used to solve the problem of the inverted pendulum - inverted pendulum.
But I've seen that people have also used machine learning techniques to solve ...
0
votes
0answers
16 views
How is the general return-based off-policy equation derived?
I'm wondering how is the general return-based off-policy equation in Safe and efficient off-policy reinforcement learning derived
$$\mathcal{R} Q(x, a):=Q(x, a)+\mathbb{E}_{\mu}\left[\sum_{t \geq 0} \...
0
votes
0answers
8 views
Training methods for bipedal robot
I am looking to train a bipedal robot using unity as a scape with a genetic algorithm. I will import the CAD into unity so the hardware is exact. My questions:
Is Unity physics accurate enough to ...
0
votes
0answers
24 views
Can someone explain how and why actor-critic networks are created this way?
Deep Deterministic Policy Gradients (DDPG) and stable Baseline Code is presented here.
The actor-critic networks are created as follows:
...
2
votes
3answers
64 views
In adversarial machine learning, how does an attacker have access to the test and training dataset in order to poison it?
In the field of adversarial machine learning, machine learning models are vulnerable to attacks both on the test and training data set. However, how does the attacker get access to these datasets? How ...
3
votes
1answer
39 views
Purpose of using actor-critic algorithms under deterministic MDP dynamics?
One of the main disadvantages of the MC Policy Gradient algorithm (REINFORCE) as described say here is the fact that it has high variance (returns, which we sample, will significantly vary from ...
3
votes
0answers
24 views
Ideas on a network that can translate image differences into motor commands?
I'd like to design a network that gets two images (an image under construction, and an ideal image), and has to come up with an action vector for a simple motor command which would augment the image ...
1
vote
1answer
31 views
How can I use deep reinforcement learning for vehicle rerouting in SUMO?
I want to use deep reinforcement learning for vehicle rerouting in SUMO, but I don't know how to start training the model.
I've already created road network and vehicle routing in SUMO-XML files (...
1
vote
1answer
22 views
Can I use my previous estimate of the state-action values as initialisation in GLIE-Monte Carlo Control?
I am trying to implement a tabular-based GLIE Monte-Carlo learning algorithm.
So I repeat n times:
create observations using my previous policy $\pi_{n-1}(s)$
update my state-action values using ...
3
votes
2answers
80 views
What is the difference between reinforcement learning and AutoML?
My vague understanding of reinforcement learning (RL) is that it's very similar to supervised learning except that it updates on a continuous feed of data/activity, this to me sounds very similar to ...
1
vote
1answer
73 views
Can you explain to me this code written in Tensorflow? [closed]
I have found this part of code, but I do not actually know how it works. Because I am new to Tensorflow, I do not know it. Can anybody help me and explain it to me?
...
1
vote
1answer
55 views
How is the policy gradient's derivative work?
I am trying to understand policy gradient method using a pytorch implementation and this tutorial.
My first question is about the end result of this gradient derivation,
mainly in this equation
$\...
2
votes
1answer
105 views
How is gradient being calculated in Andrej Karpathy's pong code?
I was going through the code by Andrej Karpathy on reinforcement learning using a policy gradient. I have some questions from the code.
Where is the logarithm of the probability being calculated? ...
3
votes
1answer
39 views
Sutton & Barto's notation $V_{t+n}$ in Chapter 7: $n$-step Bootstrapping
Until Chapter 6 of Sutton & Barto's book on Reinforcement Learning, the authors use $V$ for the current estimate of a state value. Equation (6.1), for example, shows:
$$ V(S_t) \leftarrow V(S_t) +...
