Questions tagged [reinforcement-learning]

For questions related to learning controlled by external positive reinforcement or negative feedback signal or both, where learning and use of what has been thus far learned occur concurrently.

Filter by
Sorted by
Tagged with
1
vote
0answers
20 views

Why I got the same action when I train A2C when I increase the number of episodes?

I'm working on an advantage actor-critic (A2C) reinforcement learning model but the problem when I trained the system for 3500 episodes, I start to get the same action for all my testing results. ...
1
vote
0answers
27 views

Reinforcement learning CNN input weakness

I'm trying to train a network to navigate a 48x48 2D grid, and switch pixels from on to off or off to on. The agent receives a small reward if correct, and small punishment if incorrect pixel plotted. ...
3
votes
1answer
50 views

Why Monte Carlo epsilon-soft approach cannot compute $\max Q(s,a)$?

I am new to Reinforcement learning and am currently reading up on the estimation of Q $\pi(s, a)$ values using MC epsilon-soft approach and chanced upon this algorithm. The link to the algorithm is ...
0
votes
1answer
24 views

Why is this deep Q agent constantly learning just one action?

I'm trying to implement deep q learning in the OpenAI's gym "Taxi-v3" environment. But my agent only learns to do one action in every state. What am I doning wrong? Here is the Github repository with ...
0
votes
0answers
10 views

Why we multiply probabilities with support to obtain Q-values in Distributional C51 algorithm?

In 'Deep Reinforcement Learning Hands-On' book and chapter about Distributional C51 algorithm I'm reading, that to obtain Q-values from the distribution I need to calculate the weighted sum of the ...
0
votes
1answer
19 views

What is the difference between batches in deep Q learning and supervised learning?

How is the batch loss calculated in both DQNs and simple classifiers? From what I understood, in a classifier, a common method is that you sample a mini-batch, calculate the loss for every example, ...
3
votes
1answer
33 views

How can a DQN backpropagate its loss?

I'm currently trying to take the next step in deep learning. I managed so far to write my own basic feed-forward network in python without any frameworks (just numpy and pandas), so I think I ...
1
vote
1answer
16 views

NoisyNet DQN with default parameters not exploring

I implemented a DQN algorithm that plays OpenAIs Cartpole environment. The NN architecture consists of 3 normal linear layers that encode the state, and one noisy linear layer, that predicts the Q ...
3
votes
1answer
66 views

Why is the average reward plot for my reinforcement learning agent different than the usual plots?

I'm building an RL agent using SARSA and Q-Learning for testing its capabilities. The environment is a 10x10 grid, where it gets a reward of 1 if he reaches the goal while he takes -1 every time he ...
2
votes
0answers
20 views

AlphaZero: updating policy & choosing move

I’ve been doping some research on the principles behind AlphaZero. Especially this ‘cheat sheet’(1) and this implementation(2) (in Connect4) were very useful. Yet, I still have two important ...
3
votes
1answer
34 views

Optimal RL function approximation for TicTacToe game

I modeled the TicTacToe game as a RL problem - with an environment and an agent. At first I made an "Exact" agent - using the SARSA algorithm, I saved every unique state, and chose the best (...
2
votes
1answer
62 views

What is the difference between Sutton's and Levine's REINFORCE algorithm?

I followed the videos/slides of Berkley RL course, but now I am a bit confused when implementing it. Please see the picture below. In particular, what does $i$ represent in the REINFORCE algorithm? ...
3
votes
1answer
42 views

What does the notation sup dist mean in distributional RL?

I'm trying to understand distributional RL, based on this article. In one of the equations, there is a symbol $\operatorname{sup dist}$. \begin{align} \operatorname{sup dist}_{s, a} (R(s, a) + \...
1
vote
0answers
37 views

Is it a good idea to apply reinforcement learning to dots and boxes? [closed]

I am currently in college, and trying to learn reinforcement learning by myself. My primary goal is building an agent that play games such as dots and boxes. I have sufficient highschool maths ...
3
votes
1answer
45 views

Is the Q value updated at every episode?

I trying to understand the Bellman equation for updating the Q table values. The concept of initially updating the value is clear to me. What is unclear is the subsequent updates to the value. Is the ...
2
votes
0answers
25 views

Can we use imitation learning for on-policy algorithms?

Imitation learning uses experiences of an (expert) agent to train another agent, in my understanding. If I want to use an on-policy algorithm, for example, Proximal Policy Optimization, because of it'...
2
votes
0answers
21 views

What are some approaches to estimate the transition and observation probabilities in POMDP?

What are some common approaches to estimate the transition or observation probabilities, when the probabilities are not exactly known? When realizing a POMDP model, the state model needs additional ...
3
votes
1answer
54 views

What is the intuition behind the TD(0) equation with average reward, and how is it derived?

In chapter 10 of Sutton and Barto's book (2nd edition) is given the equation for TD(0) error with average reward (equation 10.10): $$\delta_t = R_{t+1} - \bar{R} + \hat{v}(S_{t+1}, \mathbf{w}) - \hat{...
1
vote
0answers
18 views

Hyperparameter optimisation over entire range or shorter range of training episodes in Deep Reinforcement Learning

I am optimising hyperparameters for my deep reinforcement learning project (using PPO2, DQN and A2C) and was wondering: Should I find the optimum hyperparameters to get maximum reward from training ...
2
votes
0answers
27 views

How to deal with nonstationary rewards in asymmetric self-play reinforcement learning?

Suppose we're training two agents to play an asymmetric game from scratch using self play (like Zerg vs. Protoss in Starcraft). During training one of the agents can become stronger (discover a good ...
1
vote
0answers
15 views

Flattened vector observation or convolutional neural network input?

This is more of a general question of how to model/preprocess 'visual' state-observations to an Agent in Reinforcement Learning that I'll illustrate with an example. Say you have a reinforcement ...
1
vote
0answers
37 views

Reinforcement learning possible with big action space?

I’m experimenting with reinforcement learning for a 2D pixel plotting task, and am running into an issue that (I think) has to do with the big action space. It goes like this: The Agent gets two ...
1
vote
1answer
36 views

How does reinforcement learning with video data work?

Gday, so my goal is to train an agent to play MarioKart on the Nintendo DS. My first approach (in theory) was to setup an emulator on my pc and let the agent play for ages. But then a colleague ...
1
vote
0answers
11 views

How to perform Interpretability analysis toward a simple reinforcement learning network

We are currently using a RL network with the following simple structure to train a model which helps to solve a transformation task: Environment (a binary file) + reward ---> LSTM (embedding) --> FC ...
3
votes
0answers
30 views

Reinforcement Learning on quantum circuit

I am trying to teach an agent to make any random 1-qubit state reach uniform superposition. So basically, the full circuit will be ...
0
votes
0answers
26 views

Reinforcement learning without trajectories

Does it make sense to use Reinforcement Learning methods in an environment that does not have trajectories? I have a lot of states and actions in my environment. However, there are no trajectories. ...
3
votes
1answer
41 views

What's the value of making the RL agent's output stochastic opposed to deterministic?

I have a question about a reinforcement learning problem. I'm training an agent to add or delete pixels in a [12 x 12] 2D space (going to be 3D in the future). Its action space consists of two ...
5
votes
1answer
76 views

Why cannot an AI agent adjust the reward function directly?

In standard Reinforcement Learning the reward function is specified by an AI designer and is external to the AI agent. The agent attempts to find a behaviour that collects higher cumulative discounted ...
8
votes
1answer
358 views

Why is reinforcement learning not the answer to AGI?

I previously asked a question about How can an AI freely make decisions on a network?. I got a great answer about how current algorithms lack agency. The first thing I thought of was reinforcement ...
0
votes
0answers
26 views

Should I consider mean or sampled value for action selection in ppo algorithm?

When considering the policy network in PPO algorithm, we need to fit a Gaussian distribution to the neural network output (for a continuous action space problem). When I use this network to obtain ...
0
votes
0answers
14 views

Deciding std. deviation for policy network output?

When I try to fit a Normal Distribution to the output of a policy network, for a continuous action space problem, what should be its standard deviation? mean for the distribution will directly be the ...
0
votes
0answers
27 views

How would you differentiate between different on-policy reinforcement learning algorithms?

How would you differentiate between different on-policy reinforcement learning algorithms?
1
vote
0answers
30 views

Should an RL agent directly observe the reward?

I am training an A2C reinforcement learning agent in a dense reward environment (where rewards are known and explicit at every timestep). Is it redundant to include the previous reward in the current ...
1
vote
0answers
45 views

Reinforcement learning for a 2D game involving two players

I'd like to create an AI for a 2D game involving two players fighting against each other. The map look something like this (The map is a NxN array somehow randomly generated): Basically the players ...
3
votes
1answer
41 views

Does using the softmax function in Q learning not defeat the purpose of Q learning?

It is my understanding that, in Q-learning, you are trying to mimic the optimal $Q$ function $Q*$, where $Q*$ is a measure of the predicted reward received from taking action $a$ at state $s$ so that ...
1
vote
1answer
47 views

Are there OpenAI Gym continuing environments (other than inverted pendulum) and baselines?

I would like to use OpenAI Gym to solve a continuing environment, that is, a problem with a single, never-ending episode (please note I don't mean a continuous environment with continuous state and ...
2
votes
2answers
97 views

what will i be able to do in the end of AI: modern approach? [closed]

i just started the book and i was wondering , what will i be able to do in AI by the end of the book ? and more particularly, what is my position with Reinforcement Learning, deep neural networks and ...
0
votes
1answer
48 views

Designing a reinforcement learning AI for a game of connect 4

I've made a connect 4 game in javascript, and I want to design an AI for it. I made a post the other day about what output would be needed, and I think I could use images of the board and a CNN. I did ...
2
votes
1answer
48 views

Does adding a constant to all rewards change the set of optimal policies in episodic tasks?

I'm taking a Coursera course on Reinforcement learning. There was a question there that wasn't addressed in the learning material: Does adding a constant to all rewards change the set of optimal ...
3
votes
1answer
44 views

Why is the stationary distribution independent of the initial state in the proof of the policy gradient theorem?

I was going through the proof of the policy gradient theorem here: https://lilianweng.github.io/lil-log/2018/04/08/policy-gradient-algorithms.html#svpg In the section "Proof of Policy Gradient ...
1
vote
1answer
49 views

Building 'evaluation' neural networks for go, reversi, checkers etc, how to train?

I'm trying to build neural networks for games like Go, Reversi, Othello, Checkers, or even tic-tac-toe, not by calculating a move, but by making them evaluate a position. The input is any board ...
2
votes
0answers
43 views

What form of output would be needed to train a model on a connect 4 AI?

I've had a big interest in machine learning for a while, and I've followed along a few tutorials, but have never made my own project. After losing many games of connect 4 with my friends, I decided to ...
0
votes
0answers
27 views

What is the difference between value iteration and policy iteration? [duplicate]

In reinforcement learning, what is the difference between policy iteration and value iteration? As much as I understand, in value iteration, you use the Bellman equation to solve for the optimal ...
2
votes
1answer
42 views

What is the effect of picking action deterministicly at inference with Policy Gradient Methods?

In policy gradient methods such as A3C/PPO, the output from the network is probabilities for each of the actions. At training time, the action to take is sampled from the probability distribution. ...
1
vote
1answer
52 views

Where does entropy enter in Soft Actor-Critic?

I am currently trying to understand SAC (Soft Actor-Critic), and I am thinking of it as a basic actor-critic with the entropy included. However, I expected the entropy to appear in the Q-function. ...
2
votes
0answers
25 views

How would one develop an action space for a game that is proprietary?

I'm currently trying to develop an RL that will teach itself to play the popular fighting game "Tekken 7". I initially had the idea of teaching it to play generally- against actual opponents with ...
0
votes
0answers
36 views

Can supervised learning be used to solve the inverted pendulum problem?

I know that reinforcement learning has been used to solve the inverted pendulum problem. Can supervised learning be used to solve the inverted pendulum problem? For example, there could be an ...
0
votes
0answers
22 views

Bandits with missing contexts

Say I learn an optimal policy $\pi(a|c)$ for a contextual multi-armed bandit problem, where the context c is a composite of multiple context variables $c = c_1, c_2, c_3$. For example, the context is ...
4
votes
0answers
55 views

Unable to train Coach for Banana-v0 Gym environment

I have just started playing with Reinforcement learning and starting from the basics I'm trying to figure out how to solve Banana Gym with coach. Essentially ...
0
votes
0answers
28 views

How can we make sure, how well the reinforcement learning works?

I read a paper which is about Deep Reinforcement Learning and it tries to use this method on stock data set. It has been showed that it reach the maximum return(profit). It has been implemented in ...