All Questions
Tagged with hyper-parameters reinforcement-learning
19 questions
2
votes
1
answer
422
views
What's the architecture and size of neural-network-based reward models as used in reinforcement learning by human feedback
My rough understanding of RLHF as used for ChatGPT in a nutshell is this:
A reward model is trained using comparisons of different responses
to the same prompt. Human trainers rank these responses ...
1
vote
1
answer
130
views
Are there any guidelines on picking hyperparameters for Deep Reinforcement Learning?
I am trying to learn machine learning from Andrew NG's Machine learning specialization on Coursera. In the chapter about reinforcement learning Andrew NG said that if you do not select correct ...
0
votes
1
answer
241
views
Which paper describes the effect of learning_starts in Reinforcement Learning?
I have seen many popular RL libraries have a learning_start parameter. This allows the agent to collect enough experiences before training on the replay_buffer. However, I am unable to find the paper ...
- 670
1
vote
1
answer
399
views
Is the described Q-table considered large?
I never saw any rule of thumb as to what size is said as large for a q-table but I have a Q-table with like 2500 entries. Is it considered large for a tabular approach? Anyone from experience can ...
3
votes
2
answers
3k
views
How many training steps does it usually take to train an RL model?
This is my model average rewards as follow image.
How to tell if it is undertrained or not convergent? How many training steps does it usually take to train an RL model?
And I'm using PPO to train.
- 303
4
votes
1
answer
988
views
Why is a large replay buffer inefficient?
Open AI spin up says
... the replay buffer should be large enough to contain a wide range
of experiences, but it may not always be good to keep everything. If
you only use the very-most recent data, ...
- 141
2
votes
1
answer
655
views
For continuing tasks, is the choice of episode length completely arbitrary?
Let's say I'm training a reinforcement learning agent to act in some environment that perpetually continues to give the agent opportunities to earn rewards, and there is no cap on the score and there ...
- 362
2
votes
1
answer
111
views
What components of reinforcement learning influence the result the most?
I'm working on my thesis concerning a reinforcement learning problem and am trying to prioritise my time on different components of it:
Formalising the agent environment (like the design of state-, ...
- 119
4
votes
0
answers
3k
views
Optimal episode length in reinforcement learning
I have a custom environment for stock trading where an episode can be as long as 2000-3000 steps. I've run several experiments with td3 and sac algorithms, average reward per episode flattens after ...
- 361
7
votes
2
answers
4k
views
What are the best hyper-parameters to tune in reinforcement learning?
Obviously, this is somewhat subjective, but what hyper-parameters typically have the most significant impact on an RL agent's ability to learn? For example, the replay buffer size, learning rate, ...
- 293
2
votes
1
answer
182
views
For episodic tasks with an absorbing state, why can't we both have $\gamma=1$ and $T= \infty$ in the definition of the return?
For episodic tasks with an absorbing state, why can't $\gamma=1$ and $T= \infty$?
In Sutton and Barto's book, they say that, for episodic tasks with absorbing states that becomes an infinite sequence, ...
- 805
3
votes
1
answer
5k
views
What should the value of epsilon be in the Q-learning?
I am trying to understand Reinforcement Learning and already explored different Youtube videos, blog posts, and Wikipedia articles.
What I don't understand is the impact of $\epsilon$. What value ...
- 373
3
votes
1
answer
3k
views
How should I choose the target's update frequency in DQN?
I have been dealing with a problem that I'm trying to solve with DQN. A general question that I have is regarding the target's update frequency. How should it change? Depending on what factor do we ...
4
votes
1
answer
1k
views
What made your DDPG implementation on your environment work?
I am working on scheduling problem that has inherent randomness. The dimensions of action and state spaces are 1 and 5 respectively.
I am using DDPG, but it seems extremely unstable, and so far it ...
- 242
1
vote
0
answers
614
views
Why is the $\epsilon$ hyper-parameter (in the $\epsilon$-greedy policy) annealed smoothly?
As far as I understand, RL is a process that can be divided into 2 stages:
Exploring a wide range of paths (acting randomly)
Refining the current optimal paths (revolving around actions with a so-...
- 280
15
votes
2
answers
17k
views
How large should the replay buffer be?
I'm learning DDPG algorithm by following the following link: Open AI Spinning Up document on DDPG, where it is written
In order for the algorithm to have stable behavior, the replay buffer should ...
- 351
7
votes
1
answer
3k
views
Should I be decaying the learning rate and the exploration rate in the same manner?
Should I be decaying the learning rate and the exploration rate in the same manner? What's too slow and too fast of an exploration and learning rate decay? Or is it specific from model to model?
- 101
3
votes
1
answer
201
views
What is the pros and cons of increasing and decreasing the number of worker processes in A3C?
In A3C, there are several child processes and one master process. The child precesses calculate the loss and backpropagation, and the master process sums them up and updates the parameters, if I ...
- 1,077
3
votes
1
answer
1k
views
Why can't my implementation of the Actor-Critic algorithm achieve good results in the 2048 game?
I implemented the Actor-Critic with n-step TD prediction to learn to play the 2048 game
For the environment, I don't use this 2048 implementation. I use a simple one without any graphical interface, ...
- 702