Questions tagged [reinforcement-learning]

For questions related to learning controlled by external positive reinforcement or negative feedback signal or both, where learning and use of what has been thus far learned occur concurrently.

Filter by
Sorted by
Tagged with
1
vote
1answer
39 views

Should I be trying to create a generic or specific (to particular game) reinforcement learning agent?

I'm creating an RL application for the game Connect Four. In general, should I be aiming to create an application that's more generic, which would 'learn' different games, or specific to a ...
2
votes
0answers
17 views

How should I define the reward function for the Connect Four game?

I'm creating an RL application for the game Connect Four. I've researched the different strategies for the game and which positions are more favourable to lead to a win. Should I be assigning ...
1
vote
0answers
27 views

Why can the core reinforcement learning algorithms be applied to POMDPs?

Why can an AI, like AlphaStar, work in StarCraft, although the environment is only partially observable? As far as I know, there are no theoretical results on RL in the POMDP environment, but it ...
2
votes
2answers
41 views

Why is there an expectation sign in the Bellman equation?

In chapter 3.5 of Sutton's book, the value function is defined as: Can someone give me some clarification about why there is the expectation sign behind the entire equation? Considering that the ...
1
vote
0answers
21 views

How can I find the appropriate reward value for my reinforcement learning problem?

I am wondering how can I find the appropriate reward value for each specific problem. I know this is a highly empirical process, but I am sure that the value is not set totally at random. I want to ...
1
vote
0answers
11 views

Is the TD-residual defined for timesteps $t$ past the length of the episode?

Let $\mathcal{S}$ be the state-space in a reinforcement learning problem where rewards are in $\mathbb{R}$, and let $V:\mathcal{S} \to \mathbb{R}$ be an approximate value function. Following the GAE ...
2
votes
2answers
46 views

Should I always start from the same start state in reinforcement learning?

In an episodic training of an RL agent, should I always start from the same initial state or I can start from several valid initial states? For example, in a gym environment, should my ...
1
vote
0answers
37 views

Efficient algorithm to obtain near optimal policies for an MDP

Given a discrete, finite Markov Decision Process (MDP) with its usual parameters $(S, A, T, R, \gamma)$, it is possible to obtain the optimal policy $\pi^{*}$ and the optimal value function $V^{*}$ ...
1
vote
0answers
13 views

What are the most common non-Markov RL paradigms?

I am interested in doing model-free RL but not using the Markov assumptions typical for MDPs or POMDPs. What are alternative paradigms that don't rely on the Markov assumptions? Are there any common ...
0
votes
0answers
25 views

How should I define the loss function when using DQN to estimate the probability density?

I'm doing a Deep Q-learning project. All of my rewards are positive and there are two terminal states. One of them has a zero reward and the other has a high positive reward. The rewards are ...
1
vote
1answer
36 views

Can this be a possible deep q learning pseudocode?

I am not using replay here. Can this be a possible deep q learning pseudocode? ...
1
vote
0answers
22 views

How to define observation and action space for an array-like input?

I am working on a problem, and I want to implement it as a reinforcement learning problem and integrate it into the OpenAI's gym. My states are in the form of lists of length $n$, where each element ...
1
vote
0answers
12 views

How can I add logic for invalid moves when using stable-baselines in OpenAI's gym?

I want to integrate my environment into the OpenAI's gym and then use the stable baselines library for training it. The learning method in the stable baseline is with one-line learning and you don't ...
1
vote
0answers
21 views

How should I avoid illegal states in OpenAI's gym?

I'm trying to make a gym environment for a simulation problem. In my gym environment, I have a set of illegal states which I don't want my agent to go into them. What is the easiest way to add such ...
2
votes
0answers
36 views

How much time does it take to train DQN on Atari environment?

I am trying to build a DQN model for the Atari Pong game, but I am not sure whether the model is learning at all. I am using the architecture described in the paper Playing Atari with Deep ...
1
vote
0answers
29 views

Atari Breakout Infrastructure

This is how they describe their infrastructure in https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf. I want to implement the game of Atari Breakout. ...
2
votes
1answer
99 views

Which deep reinforcement learning algorithm is appropriate for my problem?

My task is to solve an optimization problem with deep reinforcement learning. I read about several algorithms like DQN, PPO, DDPG, and A2C/A3C but use cases always seem to be problems like video games ...
0
votes
1answer
24 views

Non-Neural Network algorithms for large state space in zero sum games

I was reading online that Tic Tac Toe has a state space of 3^9 = 19,683. From my basic understanding, this sounds too large to use with Q Learning, as the Q table would be huge? If that is the case, ...
2
votes
1answer
29 views

What does the notation ${s'\sim T(s,a,\cdot)}$ mean?

I have been seeing notations on Expectations with their respective subscripts such as $E_{s_0 \sim D}[V^\pi (s_0)] = \Sigma_{t=0}^\infty[\gamma^t\phi(s_t)]$. This equation is taken from https://ai....
6
votes
1answer
79 views

Is there any programming language designed by deep learning?

i mean Programming Language Implementation by Deep Learning / AI. I want to know this kind of information, any links and news are very welcome. sure AI drawing picture and design PCB, so why not.
1
vote
1answer
30 views

How can I develop a reinforcement learning agent that plays memory cards game?

I am new to RL, and I am thinking of doing a little project. The goal of the project is to learn an agent play the memory game with cards. I already created the program for detecting the cards on the ...
1
vote
0answers
32 views

What are the pros and cons of deep learning and machine learning to develop a trading system?

As I want to start coding a new Trading AI in this year (first based on Python and later maybe in C++) I stumbled over the following question: Today, I would like to make a pro/contra list with you ...
1
vote
0answers
11 views

Whats the correct loss function to use during deep Q-learning (discrete action space)

After playing around with normal Q-learning I have decided to switch to deep Q-learning and I have encountered this problem. As I understand, for a task with discrete action space, where there are 4 ...
3
votes
1answer
44 views

What does a joint probability density function have to do with Stochastic Optimal Control and Reinforcement Learning?

I stumbled upon a job offer from a company that was looking for someone who was good with Reinforcement Learning (applied to finance) and something in their offer caught my eye. It goes something like ...
3
votes
1answer
38 views

Do all expert trajectories have the same starting state in apprenticeship learning?

In the apprenticeship learning algorithm described by Ng et al. in Apprenticeship Learning via Inverse Reinforcement Learning, they mention that expert trajectories come in the form of $\{s_0^i, s_1^i\...
2
votes
0answers
29 views

Why is my DQN model not getting better?

I've created a deep Q network. My model does not get better, and can't see what I'm doing wrong. I'm new to RL. Replay Memory ...
1
vote
1answer
26 views

Are there known error bounds for TD(0) with a constant learning rate?

Is there any known error bounds for the TD(0) algorithm for the value function after a finite number of iterations? $$ \Delta_t=\max_{s \in \mathcal{S}}|v_t(s)-v_\pi(s)|$$ $$v_{t+1}(s_t)=v_t(s_t)+\...
3
votes
2answers
100 views

What are some online courses for deep reinforcement learning?

What are some (good) online courses for deep reinforcement learning? I would like the course to be both programming and theoretical. I really liked David Silver's course, but the course dates from ...
3
votes
0answers
31 views

Representation of state space, action space and reward system for Reinforcement Learning problem

I am trying to solve the problem of an agent dynamically discovering(start with no information about the environment) the environment and to explore as much of the environment as possible without ...
2
votes
1answer
55 views

How to set the target for the actor in A2C?

I did a simple Actor-Critic implementation in Keras using 2 networks where the critic learns the Q-Values of every action, and the actor predicts probabilities for choosing each action. In training, ...
1
vote
0answers
39 views

Why haven't we solved the problem of bipedal walking?

This has been a mystery to me. All the walking robots look like idiots now. But we do have a lot of simulation-based results (Flexible Muscle-Based Locomotion for Bipedal Creatures ), so why can't we ...
1
vote
1answer
45 views

What is the relationship between the Q and V functions?

Suppose we have a policy $\pi$ and we use SARSA to evaluate $Q^\pi(s, a)$, where $a$ is the policy $\pi$. Can we say that $Q^\pi(s, a) = V^\pi(s)$? The reason why I think this can be the case is ...
3
votes
0answers
60 views

What are the guidelines for defining a reward function in reinforcement learning (bandit problem)?

I'm working currently on a problem and I'm using RL (bandit problem). In my system, I have an agent that chooses an action among $k$ possible actions, and a user that decides whether the agent ...
2
votes
0answers
41 views

Policy Gradient Reward Oscillation in MATLAB

I'm trying to train a Policy Gradient Agent with Baseline for my RL research. I'm using the in-built RL toolbox from MATLAB (https://www.mathworks.com/help/reinforcement-learning/ug/pg-agents.html) ...
2
votes
0answers
53 views

How to correctly implement self-play with DQN?

I have an environment where an agent faces an equal opponent, and while I've achieved OK performance implementing DQN and treating the opponent as a part of the environment, I think performance would ...
3
votes
2answers
80 views

How to calculate the advantage in policy gradient functions?

From my understanding of the REINFORCE policy gradient method, we gently nudge the probabilities of actions based on the advantages. More specifically, the positive advantages increase the ...
2
votes
1answer
38 views

How to apply hyperparameter optimization on Monte Carlo Tree Search?

I have a basic MCTS agent for the game of Hex (a turn based game). I want to tune the parameters of UCT (the Cp parameter) and the number of rollouts parameter. Where do I have to begin? The problem ...
5
votes
2answers
78 views

Why are reinforcement learning methods sample inefficient?

Reinforcement learning methods are considered to be extremely sample ineffcient. For example, in a recent Deepmind paper by Hessel et al, they showed that in order to reach human level performance on ...
2
votes
0answers
22 views

How to draw backup diagram when reward is in expectation but next state is iterated?

I am working through Sutton and Barto's RL book. So far in the text, when backup diagrams are drawn, the reward and next state are iterated together (i.e. the equations always have $\sum_{s',r}$), ...
1
vote
1answer
35 views

Why isn't the implementation of my policy evaluation for a simple MDP converging?

I am trying to code out a policy evaluation algorithm to find the $V^\pi(s)$ for all states. The following diagram below shows the MDP. In this case i let p = q = 0.5. the rewards for each states ...
3
votes
0answers
32 views

Is this a good approach to solving Atari's “Montezuma's Revenge”?

I'm new to Reinforcement Learning. For an internship, I am currently training Atari's "Montezuma's Revenge" using a double Deep Q-Network with Hindsight Experience Replay (HER). HER is supposed to ...
2
votes
1answer
45 views

What is the purpose of argmax in the PPO algorithm?

I'm kinda new to machine learning and still not too solid on math and particularly calculus. I'm currently trying to implement PPO algorithm as described in the spiningUp website : This line is ...
2
votes
1answer
35 views

How does the optimization process in hindsight experience replay exactly work?

I was reading the following research paper Hindsight Experience Replay. This is the paper that introduces a concept called Hindsight Experience Replay (HER), which basically attempts to alleviate the ...
3
votes
1answer
40 views

How can the policy iteration algorithm be model-free if it uses the transition probabilities?

I'm actually trying to understand the policy iteration in the context of RL. I read an article presenting it and, at some point, a pseudo-code of the algorithm is given : What I can't understand is ...
4
votes
1answer
37 views

How is the Markovian property consistent in reinforcement learning based scheduling?

In Reinforcement Learning, an MDP model incorporates the Markovian property. A lot of scheduling applications in a lot of disciplines use reinforcement learning (mostly deep RL) to learn scheduling ...
2
votes
0answers
34 views

Should the RL agent be trained in an environment with real-world data or with a synthetic model?

I want to train a reinforcement learning agent in an environment with parameters (for example, the wind speed, sun irradiation, etc.) that change over time. I have recorded a limited amount of data ...
1
vote
1answer
32 views

How to let an RL Agent move the mouse?

Gday guys, I'am building a game enviroment (picture) where an agent should position the mouse on the screen (cords upper right corner) and then click to shoot a canonball. If the goal (left) is hit. ...
0
votes
1answer
35 views

How to incentivise snake to go straight to apple?

I have made a Deep Q Network for the game snake but unfortunately, the snake exhibits some unwanted behavior. It generally does quite well but sometimes it gets stuck in an infinite loop that it can't ...
3
votes
0answers
43 views

How to deal with approximate states when doing path planning?

If one is interested in implementing a path planning algorithm that is grid-based, one needs to consider the fact that your grid points will never represent the true state of the robot. How is this ...
3
votes
1answer
40 views

Can experience replay be used for training after completing every single epoch?

The DQN implements replay memory. Based on my research, I believe the replay memory starts to get used for training once there is enough experience in the memory buffer. This means the neural network ...

1
3 4
5
6 7
22