All Questions
843 questions
1
vote
1
answer
24
views
How do neural scaling laws explain the improvements from LSTMs to Transformer based models
I was reading about a study on neural scaling laws from 2017 and they noted this as a summary. From Hestness, Joel; Narang, Sharan; Ardalani, Newsha; Diamos, Gregory; Jun, Heewoo; Kianinejad, Hassan; ...
- 279
0
votes
0
answers
37
views
When should you use a transformer and when LSTM, GRU and other Neural Networks?
There is a lot of information on the Internet that the transformer is better than RNN in everything, but is it true?
Examples:
«What if I need to translate words?»
«Generate text, images?»
«Play a ...
1
vote
1
answer
36
views
RMSprop approach applied to Q-learning for adaptive dynamic learning rate
I am new to this group,
Anybody familiar with Q-learning algorithm and RMSprop approach ?
i have a question regarding the application of RMSprop approach into Q-Learning to adapt dynamically the ...
0
votes
3
answers
34
views
Why does TD3/DDPG use − 𝐸 [ 𝑄 ( 𝑠 , 𝜋 ( 𝑠 ) ) ] −E[Q(s,π(s))] as the policy loss without causing Q-values to go to infinity?
I tried to understand why TD3/DDPG use a policy loss of −E[Q(s,π(s))], which should make the policy maximize Q-values. I expected this to push Q-values to infinity over time, as there’s no explicit ...
- 19
0
votes
2
answers
55
views
Is there any actual difference between these 2 definitions of a state value function?
The definition of the value function in TRPO paper is
\begin{align}
V_\pi(s_t) &= \mathbb{E}_{a_t,s_{t+1},\ldots} \left[ \sum_{l=0}^{\infty} \gamma^l r(s_{t+l}) \right], \\[10pt]
a_t &\sim \pi(...
- 139
1
vote
0
answers
47
views
More iterations of Deep Q Learning yields worse results
As a learning exercise, I am trying to implement Q-Learning and Deep Q-learning for the two player game Dots and Boxes. I have implemented Q-Learning, and it is working great. I have one model that ...
- 11
1
vote
0
answers
28
views
Solving Highly Stochastic Environments Using Reinforcement Learning
I've been working on a reinforcement learning (RL) problem in a highly stochastic environment where the effect of the noise far outweighs the impact of the agent's actions. To illustrate, consider the ...
- 31
1
vote
0
answers
40
views
Proof for Using Q-Function in Policy Gradient Formula
Currently I am reading the OpenAI spinning up document about policy gradient and actor-critic method. In this webpage,replace Return with Action value, I think they are trying to prove that the ...
- 11
0
votes
1
answer
108
views
Does Machine Learning focus on discriminative AI while Deep Learning also focus on generative AI?
I know that Deep Learning is subset of Machine learning
But is it correct that classical ML algorithms mainly focus on implementing Discriminative AI while DL algorithms implement both Generative AI ...
- 181
0
votes
0
answers
30
views
Can a trained RL network outperforms the best training sample?
I'm working on solving a problem where I need to determine the optimal set of actions to find the path that yields the maximum reward. I'm currently using a Deep Q-Network (DQN) for this task. However,...
- 1
0
votes
1
answer
87
views
Why are there two different q-learning formulas?
I found the following q-learning formula:
in this youtube video: https://www.youtube.com/watch?v=4C133ilFm3Q&t=521s
I'm now a bit confused, since I thought, that the following one is the correct ...
- 25
2
votes
1
answer
54
views
Should the experience replay memory only contain unique experiences?
I'm training an RL agent/model (DRL/DQN).
Say that, for each learning step, the memory replay used by the agent to learn, has N elements (experiences) stored, where only X are unique elements (...
0
votes
2
answers
107
views
What do we mean by "AI is correlated"?
From Wikipedia
Causal AI is a technique in artificial intelligence that builds a causal model and can thereby make inferences using causality rather than just correlation. One practical use for ...
- 117
1
vote
2
answers
121
views
tic-tac-toe - tabular q-learning - what is the formula to calculate the number of entries in the q-table
i implemented the tabular q-learning algorithm for 3x3 tictactoe multiple times and everytime the number of entries in the q-table is 16,167. I wanna know how to calculate the number of 16,167. what ...
- 25
0
votes
0
answers
35
views
Q-Learning conditions for convergence and ergodicity
Q-learning is guaranteed to convergence if the learning rate satisfies the Robbins-Monro conditions and if every state-action pair is visited infinitely often.
Regarding the latter, does it mean that ...
- 253
0
votes
0
answers
48
views
Optimizing a nonlinear objective function in Deep Reinforcement Learning
I'm working on a reinforcement learning problem where the environment returns a reward pair $(r_{t+1}^{(a)}, r_{t+1}^{(b)})$. The goal is to maximize the following nonlinear objective function.
$$
E[\...
- 1
1
vote
2
answers
99
views
Why is there no exploration-optimality trade-off in Q-learning?
I have seen this sentence while reading an RL source (slide 29): "As discussed with MC-based off-policy control: avoidance of the exploration-optimality trade-off for on-policy methods."
...
- 1,115
-1
votes
1
answer
57
views
Suppose action selection is greedy. Is Q-learning then exactly the same algorithm as Sarsa?
Below is the Exercise 6.12 from Sutto-Barto and its solution (from the solution manual) but I was not able to understand it. I will be happy if one can make it clearer.
- 1,115
0
votes
1
answer
85
views
It is not clear why SARSA is on-policy but Q-learning off-policy
Here are SARSA and Q-learning from Sutton & Barto.
In these given forms, in my opinion, Q-learning is also on-policy because action selection is based on updated Q values. Where is my mistake, if ...
- 1,115
0
votes
1
answer
109
views
Can DQN learn with discrete state spaces?
For example in Cart Pole v1 gym environment the state space is continuous, but we discretize it to apply the Q-Learning algorithm because Q-Learning is a tabular method and only works with discrete ...
0
votes
1
answer
89
views
Can Q(s,a) be replaced by V(s) when certrain requirements are met? [closed]
I read this post, was thinking about it and now I have a hypothesis but I am not sure whether or not its correct.
I claim that in Q-learning $Q(s,a)$ can be replaced by V(s) when
$p(s'|a,s)$ is ...
- 161
0
votes
0
answers
69
views
Bias of multi-step Q Learning on/off policy
This is comes from cs2852023Fall, hw3.
I'm learning RL by myself and I cann't find answers related to this question.
Althrough it's from a homework, I believe it would be beneficial to solve the ...
- 77
2
votes
2
answers
556
views
Is it easier to use back-propagation or genetic algorithms to teach an artificial intelligence?
I am making a very simple neural network for a school project, and I would like to know what the best and easiest way to "teach" a neural network would be. From what I know, backpropagation ...
- 23
3
votes
1
answer
393
views
Does the DoubleDQN algorithm use a target network or two separate policies?
I've been looking for ways to improve my DQN. That is when I found the Double DQN algorithm. After looking at explanatory videos and posts, I've seen conflicting information:
The Double DQN algorithm ...
2
votes
1
answer
375
views
Q learning (DQN) strategy for a multiplayer zero-sum game
I have been looking for ways to train a Q-learning agent for a multiplayer zero-sum game (a variation of Tic-Tac-Toe in my case). I came up with a learning strategy I haven't found anywhere else, and ...
0
votes
0
answers
21
views
Are there leaderboards/tables/stats that compare inference times between close-sourced LLMs such as GPT 3.5/4 and Claude?
https://huggingface.co/spaces/optimum/llm-perf-leaderboard is great to compare inference times between LLMs but it misses close-sourced LLMs such as GPT 3.5/4 and Claude.
- 3,267
1
vote
2
answers
1k
views
What is the difference between densenet and resnet?
Is the only difference between the two how the skip connection is combined? Resnet combines skip connections through addition and Densenet through concatenating.
The Densenet paper appears to be ...
- 233
1
vote
1
answer
67
views
Expectile regression in Implicit Q-Learning
I am reading Kostrikov et al.'s "Offline Reinforcement Learning with Implicit Q-Learning" but got stuck understanding one particular transformation they use.
They describe the loss function ...
- 248
0
votes
0
answers
23
views
Summing up rewards when encountering terminal state in n-step DQN
I'm trying to implement n-step DQN using deque for n-step experience buffer and I am not sure how to handle the terminal state in calculation
For step = n = 5, Sx = state number x, T - terminal state, ...
- 1
1
vote
1
answer
98
views
Is there any advantage of genetic algorithm (or programming) over Neural Networks? [closed]
I am planning to switch from neural networks to genetic algorithms (GA) and programming (GP).
One of the main hassles of working with neural networks is that it requires a large amount of training ...
- 339
0
votes
0
answers
32
views
Tabular Q-Learning & TicTacToe
I'm currently implementing tabular q-learning for 3x3 tictactoe in python
and I'm new to RL and still have a hard time to understand RL.
Therefore, I would like to know one thing:
In (tabular) q-...
- 25
1
vote
1
answer
83
views
Tabular Q-Learning: Is a variable for "action_history" needed for backpropagating the q-value for all previous actions?
I am implementing a Tabular Q-Learning algorithm in Python and have questions regarding the use of an 'action_history' variable.
Necessity of 'action_history': In Q-Learning, the Q-value update is ...
- 25
0
votes
0
answers
57
views
How do I deal with dynamic, parameterized, action spaces?
I want to design an AI Learning Algorithm for a Student made, round based Game.
Let me first explain the Game/Environment
You have a round based HTTP Game, in which multiple Players can participate.
...
- 1
0
votes
1
answer
330
views
Can Q-learning rewards and next states be non-deterministic?
I am working in a team to develop a Q-learning based approach for hyperparameter tuning. I have a disagreement with one of my teammates on how they defined this problem. They defined it as follows:
...
0
votes
2
answers
61
views
Should I define my problem as image segmentation or detection?
I have a problem and have to decide wether it's an object detection or object segmentation problem. I want to use Yolov8 for training. We already have hundrets of images but they aren't labeled yet. ...
- 113
1
vote
0
answers
70
views
Resulting quantiles from Quantile Regression DQN
In my QR-DQN application, the resulting quantiles for a state s and action a take the form of the blue line in the figure. The ...
- 11
6
votes
2
answers
364
views
DQN arXiv 10-year anniversary: What are the outstanding problems being actively researched in deep Q-learning since 2019?
Background
As of today (12-19-2023), the arXiv submission of the original deep Q-learning approach to achieve superhuman performance on ATARI games has turned a decade old. The original approach, ...
- 1,703
0
votes
1
answer
86
views
How do I update Q-values in Q-learning when rewards may only be received after many actions?
I am working on a Q-learning system where the agent may well (and almost always) have to take many actions before a reward can be given to the agent (or more so, the notion of a reward in my context ...
- 103
1
vote
1
answer
1k
views
When to use Pruning, Quantization , Distillation and others when optimizing speed
I want to understand how to optimize models for inference speed and am seeking some advice and best practices for the same.
I am a little bit aware of the concepts of pruning, quantization, and ...
- 789
4
votes
2
answers
3k
views
What are the differences between seq2seq and encoder-decoder architectures?
I've read many tutorials online that use both words interchangeably. When I search and find that they are the same, why not just use one word since they have the same definition?
user78615
6
votes
2
answers
614
views
Does increasing the number of Q functions in Q-Learning scale?
Q-Learning (Watkins, 1989) uses a single function to estimate the value of actions and to choose the next action. Double Q-Learning (Hasselt, 2010) extends this and uses two functions which are ...
- 1,559
2
votes
2
answers
141
views
Can we solve the environment with only the linear and angular position through Q-Learning?
I'm trying to solve the cartpole-v1 gym environment with only the linear and angular position, but the mean reward of the last 100 episodes isn't greater than 20 rewards. The longest train i made was ...
1
vote
0
answers
36
views
Why slow-changing policy invalidates Double DQN approach in TD3 paper?
In the paper describing TD3 (https://arxiv.org/abs/1802.09477), the authors say that they could not effectively address the Q-learning overestimation bias by using different networks for maximizing ...
- 131
1
vote
1
answer
133
views
Why are these two implementations of the $\epsilon$-greedy policy different?
According to the book Reinforcement Learning An Introduction, the epsilon greedy policy can generally implemented as:
$$
\pi(a|s) =
\begin{cases}
\frac{\epsilon}{|A|} + 1 - \epsilon & \text{if } ...
- 195
2
votes
1
answer
541
views
What are the similarities between Q-learning and Value Iteration?
This is the explanation of value iteration in our notes where you keep applying bellman optimality equation till it stops changing and then acting greedily wrt the value function gives the optimal ...
- 23
1
vote
1
answer
64
views
Can Q-learning be used to create new creative solutions by combining different factors and characteristics?
References from Wikipedia:
https://en.wikipedia.org/wiki/Q-learning
https://en.wikipedia.org/wiki/Markov_decision_process
Q-learning can be used to create new creative solutions, combining different ...
- 267
1
vote
1
answer
56
views
In Q-learning, states need to be just X and Y positions of the agent, or a state can be several other characteristics?
For example, in this article: https://www.learndatasci.com/tutorials/reinforcement-q-learning-scratch-python-openai-gym/, which explains Q-learnig, teaches the Smartcab problem, the environment is a ...
- 267
2
votes
1
answer
60
views
Is Q-learning limited to just visual scenarios, or is it much broader and can it be used to solve non-visual problems as well?
For example, in this article: https://www.learndatasci.com/tutorials/reinforcement-q-learning-scratch-python-openai-gym/, which explains Q-learnig, teaches the Smartcab problem, it has a visual ...
- 267
1
vote
2
answers
313
views
In Q-learning, Am I the one who will define the way in which actions allow the agent to interact with the environment? And the interactions will vary?
In Q-learning, am I the one who will define the way in which actions allow the agent to interact with the environment, so that the way in which actions allow the agent to interact with the environment ...
- 267
1
vote
1
answer
212
views
Could someone give a very simple example of Q-learning in a very small environment? [closed]
I would really like to see an example of Q-learning that I could read, so that I can learn Q-learning from scratch. I read some articles on the internet, but I found it a little difficult to ...
- 267