Questions tagged [reinforcement-learning]

For questions related to learning controlled by external positive reinforcement or negative feedback signal or both, where learning and use of what has been thus far learned occur concurrently.

Filter by
Sorted by
Tagged with
-1
votes
0answers
21 views

Which algorithm to use?

When to use Genetic algirithm or any reinforcement learning When to use supervised learning How to use which is more suitable for which? I am a beginer. Actually just as a test I was ...
0
votes
0answers
12 views

What's the value of making the RL agent's output stochastic opposed to deterministic?

I have a question about a reinforcement learning problem. I'm training an agent to add or delete pixels in a [12 x 12] 2D space (going to be 3D in the future). Its action space consists of two ...
3
votes
1answer
45 views

Why cannot an AI agent adjust the reward function directly?

In standard Reinforcement Learning the reward function is specified by an AI designer and is external to the AI agent. The agent attempts to find a behaviour that collects higher cumulative discounted ...
2
votes
2answers
88 views

How to define reward function in POMDPs?

How do I define a reward function for my POMDP model? In the literature, it is common to use one simple number as a reward, but I am not sure if this is really how you define a function. Because this ...
4
votes
2answers
226 views

Why are lambda returns so rarely used in policy gradients?

I've seen monte-carlo reward $G_{t}$ used in REINFORCE and TD(0) reward $r_t + \gamma Q(s', a')$ used in vanilla actor-critic. I've never seen someone use lambda reward $G^{\lambda}_{t}$ in these ...
4
votes
1answer
267 views

Why is reinforcement learning not the answer to AGI?

I previously asked a question about How can an AI freely make decisions on a network?. I got a great answer about how current algorithms lack agency. The first thing I thought of was reinforcement ...
2
votes
3answers
63 views

How to enforce covariance-matrix output as part of the last layer of a Policy Network?

I have a continuous state space, and a continuous action space. The way I understand it, I can build a policy network which takes as input a continuous state vector and outputs both mean vector and ...
0
votes
0answers
24 views

Should I consider mean or sampled value for action selection in ppo algorithm?

When considering the policy network in PPO algorithm, we need to fit a Gaussian distribution to the neural network output (for a continuous action space problem). When I use this network to obtain ...
1
vote
0answers
37 views

Reinforcement learning for a 2D game involving two players

I'd like to create an AI for a 2D game involving two players fighting against each other. The map look something like this (The map is a NxN array somehow randomly generated): Basically the players ...
0
votes
0answers
12 views

Deciding std. deviation for policy network output?

When I try to fit a Normal Distribution to the output of a policy network, for a continuous action space problem, what should be its standard deviation? mean for the distribution will directly be the ...
0
votes
0answers
26 views
11
votes
1answer
5k views

How can policy gradients be applied in the case of multiple continuous actions?

Trusted Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) are two cutting edge policy gradients algorithms. When using a single continuous action, normally, you would use some ...
1
vote
0answers
29 views

Should an RL agent directly observe the reward?

I am training an A2C reinforcement learning agent in a dense reward environment (where rewards are known and explicit at every timestep). Is it redundant to include the previous reward in the current ...
1
vote
1answer
33 views

Are there OpenAI Gym continuing environments (other than inverted pendulum) and baselines?

I would like to use OpenAI Gym to solve a continuing environment, that is, a problem with a single, never-ending episode (please note I don't mean a continuous environment with continuous state and ...
3
votes
1answer
35 views

Does using the softmax function in Q learning not defeat the purpose of Q learning?

It is my understanding that, in Q-learning, you are trying to mimic the optimal $Q$ function $Q*$, where $Q*$ is a measure of the predicted reward received from taking action $a$ at state $s$ so that ...
5
votes
2answers
166 views

How can the importance-sampling ratio be different than zero?

In the book "Reinforcement Learning: An Introduction" (2018) Sutton and Barto define at page 102 the importance-sampling-ration as follows: $$\rho _{t:T-1}=\prod_{k=t}^{T-1}\frac{\pi(A_k|S_k)}{b(A_k|...
3
votes
1answer
30 views

What could be the cause of the drop in the reward in A3C?

The mean episodic reward is generally increasing, but it has spontaneous drops, and I'm not sure of their cause. The problem has a sparse reward, batch size=2000, <...
5
votes
1answer
211 views

Building AI from chess - data shape from simulation

Problem My problem is the following: Given 1000 wins, losses, and ties from a chess simulation I am using, what shape should each game be (I.e., sequence of moves leading to win/loss/tie) in order to ...
2
votes
1answer
213 views

How can we calculate the gradient of the Boltzmann policy over reward function?

I'm struggling with an inverse reinforcement learning problem which seems to appear quite often around the literature, yet I can't find any resources explaining it. The problem is that of calculating ...
3
votes
1answer
46 views

Convergence of Semi gradient TD(0) with non-linear function approximation

I am looking for a result that shows convergence of semi-gradient TD(0) algorithm with non-linear function approximation for on-policy prediction. Specifically, the update equation is given by (...
3
votes
1answer
42 views

Why is the stationary distribution independent of the initial state in the proof of the policy gradient theorem?

I was going through the proof of the policy gradient theorem here: https://lilianweng.github.io/lil-log/2018/04/08/policy-gradient-algorithms.html#svpg In the section "Proof of Policy Gradient ...
7
votes
1answer
369 views

What is the relationship between these two taxonomies for machine learning with neural networks?

Could you please let me know which of the following classification of Neural Network's learning algorithm is correct? The first one classifies it into: supervised, unsupervised and reinforcement ...
0
votes
1answer
42 views

Designing a reinforcement learning AI for a game of connect 4

I've made a connect 4 game in javascript, and I want to design an AI for it. I made a post the other day about what output would be needed, and I think I could use images of the board and a CNN. I did ...
2
votes
2answers
93 views

what will i be able to do in the end of AI: modern approach? [closed]

i just started the book and i was wondering , what will i be able to do in AI by the end of the book ? and more particularly, what is my position with Reinforcement Learning, deep neural networks and ...
3
votes
3answers
372 views

How to train a logical XOR with reinforcement learning?

After reading an excellent BLOG post Deep Reinforcement Learning: Pong from Pixels and playing with the code a little, I've tried to do something simple: use the same code to train a logical XOR gate. ...
1
vote
1answer
44 views

Building 'evaluation' neural networks for go, reversi, checkers etc, how to train?

I'm trying to build neural networks for games like Go, Reversi, Othello, Checkers, or even tic-tac-toe, not by calculating a move, but by making them evaluate a position. The input is any board ...
2
votes
1answer
33 views

Does adding a constant to all rewards change the set of optimal policies in episodic tasks?

I'm taking a Coursera course on Reinforcement learning. There was a question there that wasn't addressed in the learning material: Does adding a constant to all rewards change the set of optimal ...
2
votes
0answers
43 views

What form of output would be needed to train a model on a connect 4 AI?

I've had a big interest in machine learning for a while, and I've followed along a few tutorials, but have never made my own project. After losing many games of connect 4 with my friends, I decided to ...
29
votes
6answers
14k views

What's the difference between model-free and model-based reinforcement learning?

What's the difference between model-free and model-based reinforcement learning? It seems to me that any model-free learner, learning through trial and error, could be reframed as model-based. In ...
5
votes
1answer
106 views

How do I compute the variance of the return of an evaluation policy using two behaviour policies?

Suppose there is an evaluation policy called $\pi_{e}$ and there are two behavior policies $\pi_{b1}$ and $\pi_{b2}$. I know that it is possible to estimate the return of policy $\pi_{e}$ through ...
0
votes
0answers
27 views

What is the difference between value iteration and policy iteration? [duplicate]

In reinforcement learning, what is the difference between policy iteration and value iteration? As much as I understand, in value iteration, you use the Bellman equation to solve for the optimal ...
2
votes
1answer
75 views

What happens to the optimal value function if the reward is multiplied by a constant?

What happens to the optimal action-value function, $q_*$ if the reward is multiplied by a constant $c$? Is the optimal action-value function also multiplied by such a constant?
3
votes
1answer
73 views

Sampling in TRPO or PPO

In the TRPO paper, the objective to maximize is (equation 14) $$ \mathbb{E}_{s\sim\rho_{\theta_\text{old}},a\sim q}\left[\frac{\pi_\theta(a|s)}{q(a|s)} Q_{\theta_\text{old}}(s,a) \right] $$ which ...
3
votes
0answers
32 views

Is there any open source implementation of the SBEED learning algorithm?

Are there are any openly available implementations of the SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation paper?
2
votes
1answer
42 views

What is the effect of picking action deterministicly at inference with Policy Gradient Methods?

In policy gradient methods such as A3C/PPO, the output from the network is probabilities for each of the actions. At training time, the action to take is sampled from the probability distribution. ...
1
vote
1answer
47 views

Where does entropy enter in Soft Actor-Critic?

I am currently trying to understand SAC (Soft Actor-Critic), and I am thinking of it as a basic actor-critic with the entropy included. However, I expected the entropy to appear in the Q-function. ...
0
votes
0answers
34 views

Can supervised learning be used to solve the inverted pendulum problem?

I know that reinforcement learning has been used to solve the inverted pendulum problem. Can supervised learning be used to solve the inverted pendulum problem? For example, there could be an ...
2
votes
0answers
24 views

How would one develop an action space for a game that is proprietary?

I'm currently trying to develop an RL that will teach itself to play the popular fighting game "Tekken 7". I initially had the idea of teaching it to play generally- against actual opponents with ...
1
vote
1answer
38 views

How to properly optimize shared network between actor and critic?

I'm building an actor-critic reinforcment learning algorithm to solve environments. I want to use a single encoder to find representation of my environment. When I share the encoder with the actor ...
0
votes
0answers
22 views

Bandits with missing contexts

Say I learn an optimal policy $\pi(a|c)$ for a contextual multi-armed bandit problem, where the context c is a composite of multiple context variables $c = c_1, c_2, c_3$. For example, the context is ...
4
votes
0answers
54 views

Unable to train Coach for Banana-v0 Gym environment

I have just started playing with Reinforcement learning and starting from the basics I'm trying to figure out how to solve Banana Gym with coach. Essentially ...
0
votes
0answers
28 views

How can we make sure, how well the reinforcement learning works?

I read a paper which is about Deep Reinforcement Learning and it tries to use this method on stock data set. It has been showed that it reach the maximum return(profit). It has been implemented in ...
1
vote
0answers
23 views

How to set the multiple continuous actions with constraints

I want to build a Deep Reinforcement Learning Model for Asset allocation. Background: I have 7 stock indexes from different markets, and I want to build a policy to produce the action (likes whether ...
1
vote
1answer
25 views

Deepmind Spriteworld run_demo.py not found

I'm trying to run the Deepmind Spriteworld demo described on the project's GitHub page, but I'm not finding run_demo.py in the distribution and the closest sounding ...
5
votes
2answers
759 views

How do I create an AI for a two-players board game?

Goal I want to create an artificial intelligence to compete against other players in a board game. Game explanation I have a board game similar to 'snakes and ladders'. You have to get to a final ...
4
votes
0answers
44 views

Normalizing Normal Distributions in Thompson Sampling for online Reinforcement Learning

In my implementation of Thompson sampling (TS) for online Reinforcement Learning, my distribution for selecting $a$ is $\mathcal{N}(Q(s, a), \frac{1}{C(s,a)+1})$ where $C(s,a)$ is the number of times $...
2
votes
1answer
41 views

Understanding proof of lemma 1 (policy improvement bound) of the “Trust Region Policy Optimization” paper

In the Trust Region Policy Optimization paper, in Lemma 1 of Appendix A, I did not quite understand the transition from (21) from (20). In going from (20) to (21), $A^\pi(s_t, a_t)$ is substituted ...
1
vote
1answer
55 views

How is the policy gradient's derivative work?

I am trying to understand policy gradient method using a pytorch implementation and this tutorial. My first question is about the end result of this gradient derivation, mainly in this equation $\...
2
votes
0answers
23 views

What could be the cause of the drop of the total reward when using DQN to solve the cart-pole environment?

I'm trying to use DQN to solve the cart-pole environment. I have 2 networks (target and behavior). Both of them have 3 hidden layers with 24 neurons, using the ReLU activation. The loss is MSE and the ...
2
votes
1answer
42 views

REINFORCE algorithm for portfolio optimization - problem while training

I'm trying to implement the Reinforce algorithm (Monte Carlo policy gradient) in order to optimize a portfolio of 94 stocks on a daily basis (I have suitable historical data to achieve this). The idea ...