Questions tagged [reinforcement-learning]
For questions related to learning controlled by external positive reinforcement or negative feedback signal or both, where learning and use of what has been thus far learned occur concurrently.
900
questions
0
votes
0answers
4 views
How to integrate Dict space of OpenAI gym into a reinforcement learning framework?
I am implementing a gym environment and I have several input arrays as my input (different sizes). The most simple method to integrate my environment into the gym is to use the Dict space as my ...
3
votes
0answers
17 views
How is the state-visitation frequency computed in “Maximum Entropy Inverse Reinforcement Learning”?
I am trying to understand the formulation of the maximum entropy Inverse RL method by Brian Ziebart. Particularly, I am stuck on how to understand the computation of state - visitation frequencies.
...
2
votes
0answers
5 views
Equivalence between expected parameter increments in “Off-Policy Temporal-Difference Learning with Function Approximation”
I am having a hard time understanding the proof of theorem 1 presented in the "Off-Policy Temporal-Difference Learning with Function Approximation" paper.
Let $\Delta \theta$ and $\Delta \bar{\theta}...
1
vote
0answers
10 views
2
votes
0answers
24 views
How was the DQN trained to play many games?
Some people claim that DQN was used to play many Atari games. But what actually happened? Was DQN trained only once (with some data from all games) or was it trained separately for each game? What was ...
1
vote
1answer
86 views
In RL, if I assign the rewards for better positional play, the algorithm is learning nothing?
I'm creating an RL application for the game Connect Four.
If I tell the algorithm which moves/token positions will receive greater rewards, surely it's not actually learning anything; it's just a ...
2
votes
0answers
13 views
+50
How to formulate normalization/probability conditions on state-action spaces in Gym?
I intend to develop a custom environment for open-ai's gym.
My goal is for an agent to learn (among additional objectives) dividing a certain quantity drawn from a continous action space (i.e. spaces....
1
vote
0answers
10 views
What kind of artificial intelligence is this? A decentralized swarm intelligence where the input and output is split among the agents
I have an AI design for deciding the length of green and red lamps of the traffic. In my design, every crossroads has its own agent. This agent has input the amount of vehicle in each road in a single ...
0
votes
0answers
10 views
DQN is unable to learn from image data
I am trying to write a DQN model that will be able to solve OpenAI gym CartPole environment. I successfully managed to do it using scalar observation data that ...
1
vote
1answer
25 views
What is the relationship between the reward function and the value function?
To clarify it in my head, the value function calculates how 'good' it is to be in a certain state by summing all future (discounted) rewards, while the reward function is what the value function uses ...
1
vote
1answer
37 views
Should I be trying to create a generic or specific (to particular game) reinforcement learning agent?
I'm creating an RL application for the game Connect Four.
In general, should I be aiming to create an application that's more generic, which would 'learn' different games, or specific to a ...
2
votes
0answers
13 views
How should I define the reward function for the Connect Four game?
I'm creating an RL application for the game Connect Four. I've researched the different strategies for the game and which positions are more favourable to lead to a win.
Should I be assigning ...
1
vote
0answers
25 views
Why can the core reinforcement learning algorithms be applied to POMDPs?
Why can an AI, like AlphaStar, work in StarCraft, although the environment is only partially observable? As far as I know, there are no theoretical results on RL in the POMDP environment, but it ...
2
votes
2answers
35 views
Why is there an expectation sign in the Bellman equation?
In chapter 3.5 of Sutton's book, the value function is defined as:
Can someone give me some clarification about why there is the expectation sign behind the entire equation? Considering that the ...
1
vote
0answers
19 views
How can I find the appropriate reward value for my reinforcement learning problem?
I am wondering how can I find the appropriate reward value for each specific problem. I know this is a highly empirical process, but I am sure that the value is not set totally at random. I want to ...
1
vote
0answers
10 views
Is the TD-residual defined for timesteps $t$ past the length of the episode?
Let $\mathcal{S}$ be the state-space in a reinforcement learning problem where rewards are in $\mathbb{R}$, and let $V:\mathcal{S} \to \mathbb{R}$ be an approximate value function. Following the GAE ...
1
vote
2answers
38 views
Should I always start from the same start state in reinforcement learning?
In an episodic training of an RL agent, should I always start from the same initial state or I can start from several valid initial states?
For example, in a gym environment, should my ...
1
vote
0answers
32 views
+100
Efficient algorithm to obtain near optimal policies for an MDP
Given a discrete, finite Markov Decision Process (MDP) with its usual parameters $(S, A, T, R, \gamma)$, it is possible to obtain the optimal policy $\pi^{*}$ and the optimal value function $V^{*}$ ...
1
vote
0answers
12 views
What are the most common non-Markov RL paradigms?
I am interested in doing model-free RL but not using the Markov assumptions typical for MDPs or POMDPs.
What are alternative paradigms that don't rely on the Markov assumptions? Are there any common ...
0
votes
0answers
18 views
How should I define the loss function when using DQN to estimate the probability density?
I'm doing a Deep Q-learning project. All of my rewards are positive and there are two terminal states. One of them has a zero reward and the other has a high positive reward.
The rewards are ...
0
votes
0answers
20 views
Is the MDP value function used in the minimax algorithm? [closed]
Is the MDP value function represented (or used) in the minimax algorithm? If so, could you please give an example?
1
vote
1answer
30 views
Can this be a possible deep q learning pseudocode?
I am not using replay here. Can this be a possible deep q learning pseudocode?
...
1
vote
0answers
21 views
How to define observation and action space for an array-like input?
I am working on a problem, and I want to implement it as a reinforcement learning problem and integrate it into the OpenAI's gym.
My states are in the form of lists of length $n$, where each element ...
1
vote
0answers
7 views
How can I add logic for invalid moves when using stable-baselines in OpenAI's gym?
I want to integrate my environment into the OpenAI's gym and then use the stable baselines library for training it.
The learning method in the stable baseline is with one-line learning and you don't ...
1
vote
0answers
20 views
How should I avoid illegal states in OpenAI's gym?
I'm trying to make a gym environment for a simulation problem. In my gym environment, I have a set of illegal states which I don't want my agent to go into them. What is the easiest way to add such ...
2
votes
0answers
21 views
How much time does it take to train DQN on Atari environment?
I am trying to build a DQN model for the Atari Pong game, but I am not sure whether the model is learning at all.
I am using the architecture described in the paper Playing Atari with Deep ...
1
vote
0answers
27 views
Atari Breakout Infrastructure
This is how they describe their infrastructure in https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf. I want to implement the game of Atari Breakout.
...
2
votes
1answer
84 views
Which deep reinforcement learning algorithm is appropriate for my problem?
My task is to solve an optimization problem with deep reinforcement learning. I read about several algorithms like DQN, PPO, DDPG, and A2C/A3C but use cases always seem to be problems like video games ...
0
votes
1answer
22 views
Non-Neural Network algorithms for large state space in zero sum games
I was reading online that Tic Tac Toe has a state space of 3^9 = 19,683. From my basic understanding, this sounds too large to use with Q Learning, as the Q table would be huge?
If that is the case, ...
2
votes
1answer
29 views
What does the notation ${s'\sim T(s,a,\cdot)}$ mean?
I have been seeing notations on Expectations with their respective subscripts such as $E_{s_0 \sim D}[V^\pi (s_0)] = \Sigma_{t=0}^\infty[\gamma^t\phi(s_t)]$. This equation is taken from https://ai....
5
votes
1answer
69 views
Is there any programming language designed by deep learning?
i mean Programming Language Implementation by Deep Learning / AI.
I want to know this kind of information, any links and news are very welcome.
sure AI drawing picture and design PCB, so why not.
1
vote
1answer
27 views
How can I develop a reinforcement learning agent that plays memory cards game?
I am new to RL, and I am thinking of doing a little project. The goal of the project is to learn an agent play the memory game with cards.
I already created the program for detecting the cards on the ...
1
vote
0answers
29 views
What are the pros and cons of deep learning and machine learning to develop a trading system?
As I want to start coding a new Trading AI in this year (first based on Python and later maybe in C++) I stumbled over the following question:
Today, I would like to make a pro/contra list with you ...
1
vote
0answers
11 views
Whats the correct loss function to use during deep Q-learning (discrete action space)
After playing around with normal Q-learning I have decided to switch to deep Q-learning and I have encountered this problem.
As I understand, for a task with discrete action space, where there are 4 ...
2
votes
1answer
42 views
What does a joint probability density function have to do with Stochastic Optimal Control and Reinforcement Learning?
I stumbled upon a job offer from a company that was looking for someone who was good with Reinforcement Learning (applied to finance) and something in their offer caught my eye. It goes something like ...
3
votes
1answer
33 views
Do all expert trajectories have the same starting state in apprenticeship learning?
In the apprenticeship learning algorithm described by Ng et al. in Apprenticeship Learning via Inverse Reinforcement Learning, they mention that expert trajectories come in the form of $\{s_0^i, s_1^i\...
1
vote
0answers
24 views
Why is my DQN model not getting better?
I've created a deep Q network. My model does not get better, and can't see what I'm doing wrong. I'm new to RL.
Replay Memory
...
1
vote
0answers
16 views
Are there known error bounds for TD(0) with a constant learning rate?
Is there any known error bounds for the TD(0) algorithm for the value function after a finite number of iterations?
$$ \Delta_t=\max_{s \in \mathcal{S}}|v_t(s)-v_\pi(s)|$$
$$v_{t+1}(s_t)=v_t(s_t)+\...
2
votes
2answers
74 views
What are some online courses for deep reinforcement learning?
What are some (good) online courses for deep reinforcement learning?
I would like the course to be both programming and theoretical. I really liked David Silver's course, but the course dates from ...
2
votes
0answers
27 views
Representation of state space, action space and reward system for Reinforcement Learning problem
I am trying to solve the problem of an agent dynamically discovering(start with no information about the environment) the environment and to explore as much of the environment as possible without ...
2
votes
1answer
30 views
How to set the target for the actor in A2C?
I did a simple Actor-Critic implementation in Keras using 2 networks where the critic learns the Q-Values of every action, and the actor predicts probabilities for choosing each action. In training, ...
0
votes
0answers
17 views
Understanding log_prob for Normal distribution in pytorch [migrated]
I'm currently trying to solve Pendulum-v0 from the openAi gym environment which has a continuous action space. As a result, I need to use a Normal Distribution to sample my actions. What I don't ...
1
vote
0answers
38 views
Why haven't we solved the problem of bipedal walking?
This has been a mystery to me.
All the walking robots look like idiots now. But we do have a lot of simulation-based results
(Flexible Muscle-Based Locomotion for Bipedal Creatures
), so why can't we ...
1
vote
1answer
43 views
What is the relationship between the Q and V functions?
Suppose we have a policy $\pi$ and we use SARSA to evaluate $Q^\pi(s, a)$, where $a$ is the policy $\pi$.
Can we say that $Q^\pi(s, a) = V^\pi(s)$?
The reason why I think this can be the case is ...
2
votes
0answers
55 views
What are the guidelines for defining a reward function in reinforcement learning (bandit problem)?
I'm working currently on a problem and I'm using RL (bandit problem).
In my system, I have an agent that chooses an action among $k$ possible actions, and a user that decides whether the agent ...
1
vote
0answers
36 views
Policy Gradient Reward Oscillation in MATLAB
I'm trying to train a Policy Gradient Agent with Baseline for my RL research. I'm using the in-built RL toolbox from MATLAB (https://www.mathworks.com/help/reinforcement-learning/ug/pg-agents.html) ...
1
vote
0answers
41 views
How to correctly implement self-play with DQN?
I have an environment where an agent faces an equal opponent, and while I've achieved OK performance implementing DQN and treating the opponent as a part of the environment, I think performance would ...
2
votes
0answers
34 views
How to calculate the advantage in policy gradient functions?
From my understanding of the REINFORCE policy gradient method, we gently nudge the probabilities of actions based on the advantages. More specifically, the positive advantages increase the ...
1
vote
0answers
23 views
How to apply hyperparameter optimization on Monte Carlo Tree Search?
I have a basic MCTS agent for the game of Hex (a turn based game). I want to tune the parameters of UCT (the Cp parameter) and the number of rollouts parameter.
Where do I have to begin? The problem ...
2
votes
2answers
58 views
Why are reinforcement learning methods sample inefficient?
Reinforcement learning methods are considered to be extremely sample ineffcient.
For example, in a recent Deepmind paper by Hessel et al, they showed that in order to reach human level performance on ...
