Questions tagged [reinforcement-learning]

For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.

Filter by
Sorted by
Tagged with
1
vote
3answers
126 views

Can people use neural networks without providing the set of training data?

It seems that neural networks (NNs) can be applied to supervised learning, unsupervised learning and reinforcement learning. Some people even train neural networks without the set of training data. If ...
0
votes
0answers
25 views

Can people set loss function of neural network by themselves instead of choosing cross entropy or mean square error?

I found people used deep neural network to get optimal policy by solving a nonconvex optimization problem. Moreover, they didn't use any set of training data and claimed that it's the difference ...
0
votes
1answer
17 views

In a DDQN architecture, why is the value of a state assumed to be the average of the Q values of the actions?

In a Dueling DQN agent (Wang et al.), the Q function is decomposed as $$ Q(s, a)=V(s) + A(s, a) - \frac{1}{|A|}\sum_{a'\in \mathcal{A}}A(s, a') $$ representing the value of the state, plus the ...
1
vote
1answer
61 views

Which policy has to be followed by a player while construction of its own Q-table?

Consider the scenario, where there are two players. One of the players perform the action randomly, whereas I want second player as a Q-player. I mean, the player selects a best action from the Q-...
0
votes
0answers
25 views

Actor-critic reinforcement learning updates and episode length

I am currently using a TD3 agent-critic network to control a vehicle suspension system, where the reward (or rather a penalty) is based on the vertical acceleration of the mass and is calculated at ...
0
votes
1answer
21 views

GLIE MC control (reinforcement learning): how the policy affects evaluation?

In his lecture 5 of the course "Reinforcement Learning", David Silver introduced GLIE Monte-Carlo Control. I understand that we do policy evaluation for one step and then policy improvement....
0
votes
0answers
20 views

How should we interpret "common coarsening" in this proof of the uniqueness of coarsest bisimulation?

On page 4 of this pdf in a theoretical RL course, we have a proof of the uniquness of the coarsest bisimulation. A bisimulation $\phi$ is a mapping from states $s \in\mathcal{S}$ to abstract states $\...
0
votes
1answer
79 views

Which policy do I need to use in updating Q function?

Policy function can be of two types: deterministic policy and stochastic policy. Deterministic policy is of the form $\pi : S \rightarrow A$ Stochastic policy is defined using conditional probability ...
1
vote
0answers
36 views

Can we combine policy evaluation and value iteration steps for solving model-based MDP?

In Sutton & Barto (2nd edition), at the very end on page 83, the following is mentioned: In general, the entire class of truncated policy iteration algorithms can be thought of as sequences of ...
1
vote
0answers
34 views

Is there a way to beat AlphaGo Zero with different method?

As I read the research from https://deepmind.com/research It seem AlphagoZero use zero knowledge and use Reinforcement learning to improve the ai skill of playing. Is there a way to beat AlphagoZero? ...
1
vote
1answer
69 views

In addition to the reward function, which other functions do I need to implement Q-learning?

In general, $Q$ function is defined as $$Q : S \times A \rightarrow \mathbb{R}$$ $$Q(s_t,a_t) = Q(s_t,a_t) + \alpha[r_{t+1} + \gamma \max\limits_{a} Q(s_{t+1},a) - Q(s_t,a_t)] $$ $\alpha$ and $\gamma$...
3
votes
1answer
74 views

What do the terms 'Bellman backup' and 'Bellman error' mean?

Some RL literature use terms such as: 'Bellman backup' and 'Bellman error'. What do these terms refer to?
1
vote
1answer
54 views

In value iteration, what happens if we try to obtain the greedy policy while looping through the states?

I am referring to the Value Iteration (VI) algorithm as mentioned in Sutton's book below. Rather than getting the greedy deterministic policy after VI converges, what happens if we try to obtain the ...
2
votes
1answer
61 views

How do I represent sample efficiency of RL rewards in mathematical notation?

I define sample efficiency as the area under the curve/graph, where $x$-axis is the number of episodes while y-axis is the cumulative reward for that episode. I would like to formally define it with a ...
0
votes
0answers
25 views

Recommended literature on layers for reinforcement learning

I was recommended to ask here after I posted on stack overflow wrongly. I was wondering if anyone had any recommended readings on layers used in neural networks for reinforcement learning? I've been ...
1
vote
1answer
83 views

How does the neural network learn when used in the REINFORCE algorithm?

As per my understanding, you run an entire episode, which contains many steps, and then back-propagate using just a single loss value. How does the neural network learn to differentiate between good ...
1
vote
1answer
20 views

Is it possible to have values of the states equal to $0$ at the end of the value iteration?

I am new to Reinforcement Learning and I am trying to self learn it. I have already posted some quesiton here and your answershave been really useful to me, so here I am posting another one. I am ...
1
vote
0answers
16 views

Are there guiding principles as to which activation functions suit a given RL algorithm?

Are there rules of thumb as to which activation functions work well (or which one would not) on the policy and value network of a class of RL algorithms? For hidden layers and for the output layer. ...
1
vote
1answer
38 views

How do we get from conditional expectation on both state and action to only state in the proof of the Policy Improvement Theorem?

I'm going through Sutton and Barto's book Reinforcement Learning: An Introduction and I'm trying to understand the proof of the Policy Improvement Theorem, presented at page 78 of the physical book. ...
1
vote
0answers
61 views

Too slow search using MCTS in OpenAI Atari games

I'm recently using Monte Carlo Tree Search in OpenAi Gym Atari, but the result isn't satisfying. Without render, the game lasts about 180 steps ( env.step() was called this much time ) with random ...
0
votes
1answer
46 views

How does CURL extract labels from logits? [closed]

While going over the pseudocode of the CURL paper, the method to identify labels from the logits wasn't clear to me. I believe this technique might be common in other PyTorch/Deep Learning tasks. I ...
2
votes
0answers
51 views

In NEAT, how do node numbers work?

I have read a lot of debates about node ids and such. I'm not 100% sure how it works, but I am assuming the next node added to a network would be the next number in that specific networks list? For ...
1
vote
2answers
220 views

What is the difference between a reward and a value for a given state?

I am trying to learn reinforcement learning and I am focusing on the value iteration. I am looking at the example of grid world, and I am trying to implement it in python. While doing this, I ...
3
votes
0answers
49 views

How does the paper implement NEAT without a global set tracking Innovations?

I have been reading this paper on NEAT and trying to implement the algorithm in C#. For the most part, I understand everything in the paper however, there are 2 things I don't understand that confuse ...
1
vote
0answers
22 views

Is there any work that applies the approach in "Finite-Sample Convergence Rates for Q-Learning and Indirect Algorithms" to standard Q-learning?

I am trying to mathematically characterize the finite sample convergence rates for Q-learning. To this end, I have read the following papers Learning rates for Q-learning, by Eyal Even-Dar et al.; ...
0
votes
1answer
39 views

How would you shape a reward function if there was four quantities to optimize?

I found this article quite useful on how to shape a reward function in RL. However, the example they gave is quite simple, where the goal is to minimize only two quantities (velocity and distance). ...
0
votes
0answers
15 views

Prioritized Experience Replay, clarifications for Important Sampling

I can't seem to understand how the weight equation is dissected and how it really works when combined with the TD-error value. The weight equation is: I can understand what N, P(i) and beta represent,...
3
votes
1answer
55 views

Are policy and value iteration used only in grid world like scenarios?

I am trying to self learn reinforcement learning. At the moment I am focusing on policy and value iteration, and I am finding several problems and doubts. One of the main doubts is given by the fact ...
2
votes
1answer
59 views

Is the Bandit Problem an MDP?

I've read Sutton and Barto's introductory RL book. They define a policy as a mapping from states to probabilities of selecting each possible action. If the agent is following policy $\pi$ at time $t$, ...
1
vote
1answer
64 views

How to enforce action bounds between 0 & 1 in soft actor-critic algorithm?

In the paper "Soft Actor-Critic Algorithms and Applications", appendix C shows enforcing action bounds using the tanh squashing function which is in (-1, 1). I have action bounds in (0, 1), ...
2
votes
1answer
33 views

What does $v(S_{t+1})$ mean in the optimal state-action value function?

In Sutton & Barto's Reinforcement Learning: An Introduction page 63 the authors introduce the optimal state value function in the expression of the optimal action-value function as follows: $q_{*}(...
1
vote
0answers
17 views

CFD Reinforcement Learning Topology optimization wind tunnel

I want to create a reinforcement learning environment, designed for win tunnel simulations, where for each iteration a deep convolutional model could receive the 3D vector/scalar fields from the past ...
1
vote
1answer
42 views

How to detect entities in Montezuma's Revenge environment

I'm thinking of implementing "Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation" paper. In this paper authors used some custom object ...
3
votes
1answer
87 views

How to prove the second form of Bellman's equation?

I'd like to prove this "second form" of Bellman's equation: $v(s) = \mathbb{E}[R_{t + 1} + \gamma v(S_{t+1}) \mid S_{t} = s]$ starting from Bellman's equation: $v(s) = \mathbb{E}[G_{t} \mid ...
0
votes
0answers
12 views

How do I quantify the difference in sample efficiency for two almost similar methods?

I am comparing my coded TD3 (Twin-Delayed DDPG) and the same TD3 (same hyperparameters) but with Priority Replay Buffer instead of a normal Replay Buffer. From what I have read, PER (Priority ...
0
votes
0answers
24 views

(Deep) feature engineering for lambda terms (mathematical expressions, higher order logic formulas) - is such thing?

Automated theorem proving with (deep) reinforcement learning (DRL) approach is hot topic in current AI research when domains of games are becoming saturated and completed research topics. For example, ...
0
votes
0answers
34 views

Bridging the gap between simulation and real-world scenarios!

I've got a DRL model that was trained on a simulation at a frame rate of 100fps, after testing it with 100fps it gives good results however when testing it with another frame rate say 50fps it gives a ...
2
votes
1answer
375 views

What is the difference between terminal state, nonterminal states and normal states?

In Sutton & Barto's Reinforcement Learning: An Introduction, page 54, the authors define the terminal state as following: Each episode ends in a special state called the terminal state But the ...
0
votes
0answers
21 views

Maximize delayed rewards

Given a Neural Network with a Dense(3) output and three actions: 'B' is [0, 0, 1] (= 1, for the sake of our example) 'N' is ...
2
votes
1answer
174 views

What are the various problems RL is trying to solve?

I have read most of Sutton and Barto's introductory text on reinforcement learning. I thought I would try to apply some of the RL algorithms in the book to a previous assignment I had done on Sokoban, ...
0
votes
0answers
37 views

How to choose the reward in reinforcement learning? [duplicate]

I am solving a combinatorial optimization problem, where I do not have a global optimum, so the goal is to improve the objective function as much as possible. So, to do this, I was inspired by this ...
0
votes
0answers
28 views

Is it possible to use Neural Networks with Contextual Bandit to learn the probability distributions instead of providing them?

I want to ask you if it's possible by using neural networks jointly with the Contextual Bandit algorithm to learn the probability distributions by which the rewards are computed as a function of the ...
0
votes
2answers
55 views

Best way to use/learn ML for board-game reinforcement learning

I am relatively new to Python but I taught myself enough to code a two-player board game that is similar to chess. It has a simple Tkinter UI. Now I am dipping into machine learning, and I want to ...
2
votes
2answers
69 views

How is it possible that Q-learning can learn a state-action value without taking into account the policy followed thereafter?

From my readings, I have been taught that the state-action value depends on the policy being followed. That seems logical because the expected return from actual actions will be different depending on ...
0
votes
1answer
49 views

Understanding "belief states" in Bayesian RL

In Bayesian RL, the uncertainty about the transition probabilities $\mathcal{P}$ parameterized by $\theta$ is captured by viewing $\theta$ as a random variable, and updating the pdf of $\theta$ after ...
0
votes
1answer
46 views

How to formulate discounted return in cartpole?

I am trying to formulate a problem that aims to prolong the lifetime of the simulation, the same as the Cartpole problem. I aware that there are two types of return: finite horizon undiscounted ...
0
votes
0answers
336 views

Optimal episode length in reinforcement learning

I have a custom environment for stock trading where an episode can be as long as 2000-3000 steps. I've run several experiments with td3 and sac algorithms, average reward per episode flattens after ...
3
votes
2answers
116 views

What are the best hyper-parameters to tune in reinforcement learning?

Obviously, this is somewhat subjective, but what hyper-parameters typically have the most significant impact on an RL agent's ability to learn? For example, the replay buffer size, learning rate, ...
1
vote
1answer
41 views

Why did Distributional Q Learning go out of popularity?

I read some papers (for example, this) and blogs that spoke about the advantages of distributional Q learning. However, it no longer seems to come up in literature. Did it have any shortcomings that ...
0
votes
0answers
40 views

DQN learns to always choose the same action for all states

I have created an RL model that uses QBased policy with a neural network for estimating Q values. My action space is of 27 actions, where each action is a 3 tuple where each value can be 1, 2 or 3. ...

1
3 4
5
6 7
38