Skip to main content

All Questions

Filter by
Sorted by
Tagged with
1 vote
1 answer
24 views

How do neural scaling laws explain the improvements from LSTMs to Transformer based models

I was reading about a study on neural scaling laws from 2017 and they noted this as a summary. From Hestness, Joel; Narang, Sharan; Ardalani, Newsha; Diamos, Gregory; Jun, Heewoo; Kianinejad, Hassan; ...
Jacob B's user avatar
  • 279
0 votes
0 answers
37 views

When should you use a transformer and when LSTM, GRU and other Neural Networks?

There is a lot of information on the Internet that the transformer is better than RNN in everything, but is it true? Examples: «What if I need to translate words?» «Generate text, images?» «Play a ...
Nikolai Vorobiev's user avatar
1 vote
1 answer
36 views

RMSprop approach applied to Q-learning for adaptive dynamic learning rate

I am new to this group, Anybody familiar with Q-learning algorithm and RMSprop approach ? i have a question regarding the application of RMSprop approach into Q-Learning to adapt dynamically the ...
Marouane Ben-akka's user avatar
0 votes
3 answers
32 views

Why does TD3/DDPG use − 𝐸 [ 𝑄 ( 𝑠 , 𝜋 ( 𝑠 ) ) ] −E[Q(s,π(s))] as the policy loss without causing Q-values to go to infinity?

I tried to understand why TD3/DDPG use a policy loss of −E[Q(s,π(s))], which should make the policy maximize Q-values. I expected this to push Q-values to infinity over time, as there’s no explicit ...
Omar's user avatar
  • 19
0 votes
2 answers
55 views

Is there any actual difference between these 2 definitions of a state value function?

The definition of the value function in TRPO paper is \begin{align} V_\pi(s_t) &= \mathbb{E}_{a_t,s_{t+1},\ldots} \left[ \sum_{l=0}^{\infty} \gamma^l r(s_{t+l}) \right], \\[10pt] a_t &\sim \pi(...
craaaft's user avatar
  • 139
1 vote
0 answers
47 views

More iterations of Deep Q Learning yields worse results

As a learning exercise, I am trying to implement Q-Learning and Deep Q-learning for the two player game Dots and Boxes. I have implemented Q-Learning, and it is working great. I have one model that ...
Joe's user avatar
  • 11
1 vote
0 answers
28 views

Solving Highly Stochastic Environments Using Reinforcement Learning

I've been working on a reinforcement learning (RL) problem in a highly stochastic environment where the effect of the noise far outweighs the impact of the agent's actions. To illustrate, consider the ...
PJORR's user avatar
  • 31
1 vote
0 answers
40 views

Proof for Using Q-Function in Policy Gradient Formula

Currently I am reading the OpenAI spinning up document about policy gradient and actor-critic method. In this webpage,replace Return with Action value, I think they are trying to prove that the ...
jim1124's user avatar
  • 11
0 votes
1 answer
108 views

Does Machine Learning focus on discriminative AI while Deep Learning also focus on generative AI?

I know that Deep Learning is subset of Machine learning But is it correct that classical ML algorithms mainly focus on implementing Discriminative AI while DL algorithms implement both Generative AI ...
DSP_CS's user avatar
  • 181
0 votes
0 answers
30 views

Can a trained RL network outperforms the best training sample?

I'm working on solving a problem where I need to determine the optimal set of actions to find the path that yields the maximum reward. I'm currently using a Deep Q-Network (DQN) for this task. However,...
Amanli's user avatar
  • 1
0 votes
1 answer
87 views

Why are there two different q-learning formulas?

I found the following q-learning formula: in this youtube video: https://www.youtube.com/watch?v=4C133ilFm3Q&t=521s I'm now a bit confused, since I thought, that the following one is the correct ...
Hans123's user avatar
  • 25
2 votes
1 answer
54 views

Should the experience replay memory only contain unique experiences?

I'm training an RL agent/model (DRL/DQN). Say that, for each learning step, the memory replay used by the agent to learn, has N elements (experiences) stored, where only X are unique elements (...
Jose Alberto Salazar's user avatar
0 votes
2 answers
107 views

What do we mean by "AI is correlated"?

From Wikipedia Causal AI is a technique in artificial intelligence that builds a causal model and can thereby make inferences using causality rather than just correlation. One practical use for ...
quanity's user avatar
  • 117
1 vote
2 answers
121 views

tic-tac-toe - tabular q-learning - what is the formula to calculate the number of entries in the q-table

i implemented the tabular q-learning algorithm for 3x3 tictactoe multiple times and everytime the number of entries in the q-table is 16,167. I wanna know how to calculate the number of 16,167. what ...
Hans123's user avatar
  • 25
0 votes
0 answers
35 views

Q-Learning conditions for convergence and ergodicity

Q-learning is guaranteed to convergence if the learning rate satisfies the Robbins-Monro conditions and if every state-action pair is visited infinitely often. Regarding the latter, does it mean that ...
Simon's user avatar
  • 253
0 votes
0 answers
48 views

Optimizing a nonlinear objective function in Deep Reinforcement Learning

I'm working on a reinforcement learning problem where the environment returns a reward pair $(r_{t+1}^{(a)}, r_{t+1}^{(b)})$. The goal is to maximize the following nonlinear objective function. $$ E[\...
Alex's user avatar
  • 1
1 vote
2 answers
99 views

Why is there no exploration-optimality trade-off in Q-learning?

I have seen this sentence while reading an RL source (slide 29): "As discussed with MC-based off-policy control: avoidance of the exploration-optimality trade-off for on-policy methods." ...
DSPinfinity's user avatar
  • 1,115
-1 votes
1 answer
57 views

Suppose action selection is greedy. Is Q-learning then exactly the same algorithm as Sarsa?

Below is the Exercise 6.12 from Sutto-Barto and its solution (from the solution manual) but I was not able to understand it. I will be happy if one can make it clearer.
DSPinfinity's user avatar
  • 1,115
0 votes
1 answer
85 views

It is not clear why SARSA is on-policy but Q-learning off-policy

Here are SARSA and Q-learning from Sutton & Barto. In these given forms, in my opinion, Q-learning is also on-policy because action selection is based on updated Q values. Where is my mistake, if ...
DSPinfinity's user avatar
  • 1,115
0 votes
1 answer
109 views

Can DQN learn with discrete state spaces?

For example in Cart Pole v1 gym environment the state space is continuous, but we discretize it to apply the Q-Learning algorithm because Q-Learning is a tabular method and only works with discrete ...
Vitor Martins's user avatar
0 votes
1 answer
89 views

Can Q(s,a) be replaced by V(s) when certrain requirements are met? [closed]

I read this post, was thinking about it and now I have a hypothesis but I am not sure whether or not its correct. I claim that in Q-learning $Q(s,a)$ can be replaced by V(s) when $p(s'|a,s)$ is ...
NMO's user avatar
  • 161
0 votes
0 answers
69 views

Bias of multi-step Q Learning on/off policy

This is comes from cs2852023Fall, hw3. I'm learning RL by myself and I cann't find answers related to this question. Althrough it's from a homework, I believe it would be beneficial to solve the ...
yeebo xie's user avatar
2 votes
2 answers
556 views

Is it easier to use back-propagation or genetic algorithms to teach an artificial intelligence?

I am making a very simple neural network for a school project, and I would like to know what the best and easiest way to "teach" a neural network would be. From what I know, backpropagation ...
AlexanderB's user avatar
3 votes
1 answer
393 views

Does the DoubleDQN algorithm use a target network or two separate policies?

I've been looking for ways to improve my DQN. That is when I found the Double DQN algorithm. After looking at explanatory videos and posts, I've seen conflicting information: The Double DQN algorithm ...
Vladislav Korecký's user avatar
2 votes
1 answer
375 views

Q learning (DQN) strategy for a multiplayer zero-sum game

I have been looking for ways to train a Q-learning agent for a multiplayer zero-sum game (a variation of Tic-Tac-Toe in my case). I came up with a learning strategy I haven't found anywhere else, and ...
Vladislav Korecký's user avatar
0 votes
0 answers
21 views

Are there leaderboards/tables/stats that compare inference times between close-sourced LLMs such as GPT 3.5/4 and Claude?

https://huggingface.co/spaces/optimum/llm-perf-leaderboard is great to compare inference times between LLMs but it misses close-sourced LLMs such as GPT 3.5/4 and Claude.
Franck Dernoncourt's user avatar
1 vote
2 answers
1k views

What is the difference between densenet and resnet?

Is the only difference between the two how the skip connection is combined? Resnet combines skip connections through addition and Densenet through concatenating. The Densenet paper appears to be ...
JobHunter69's user avatar
1 vote
1 answer
67 views

Expectile regression in Implicit Q-Learning

I am reading Kostrikov et al.'s "Offline Reinforcement Learning with Implicit Q-Learning" but got stuck understanding one particular transformation they use. They describe the loss function ...
user118967's user avatar
0 votes
0 answers
23 views

Summing up rewards when encountering terminal state in n-step DQN

I'm trying to implement n-step DQN using deque for n-step experience buffer and I am not sure how to handle the terminal state in calculation For step = n = 5, Sx = state number x, T - terminal state, ...
Question's user avatar
1 vote
1 answer
98 views

Is there any advantage of genetic algorithm (or programming) over Neural Networks? [closed]

I am planning to switch from neural networks to genetic algorithms (GA) and programming (GP). One of the main hassles of working with neural networks is that it requires a large amount of training ...
user366312's user avatar
0 votes
0 answers
32 views

Tabular Q-Learning & TicTacToe

I'm currently implementing tabular q-learning for 3x3 tictactoe in python and I'm new to RL and still have a hard time to understand RL. Therefore, I would like to know one thing: In (tabular) q-...
Hans123's user avatar
  • 25
1 vote
1 answer
83 views

Tabular Q-Learning: Is a variable for "action_history" needed for backpropagating the q-value for all previous actions?

I am implementing a Tabular Q-Learning algorithm in Python and have questions regarding the use of an 'action_history' variable. Necessity of 'action_history': In Q-Learning, the Q-value update is ...
Hans123's user avatar
  • 25
0 votes
0 answers
57 views

How do I deal with dynamic, parameterized, action spaces?

I want to design an AI Learning Algorithm for a Student made, round based Game. Let me first explain the Game/Environment You have a round based HTTP Game, in which multiple Players can participate. ...
Andre's user avatar
  • 1
0 votes
1 answer
330 views

Can Q-learning rewards and next states be non-deterministic?

I am working in a team to develop a Q-learning based approach for hyperparameter tuning. I have a disagreement with one of my teammates on how they defined this problem. They defined it as follows: ...
Ahmed Mokhtar's user avatar
0 votes
2 answers
61 views

Should I define my problem as image segmentation or detection?

I have a problem and have to decide wether it's an object detection or object segmentation problem. I want to use Yolov8 for training. We already have hundrets of images but they aren't labeled yet. ...
Ef Ge's user avatar
  • 113
1 vote
0 answers
70 views

Resulting quantiles from Quantile Regression DQN

In my QR-DQN application, the resulting quantiles for a state s and action a take the form of the blue line in the figure. The ...
amavrits's user avatar
6 votes
2 answers
364 views

DQN arXiv 10-year anniversary: What are the outstanding problems being actively researched in deep Q-learning since 2019?

Background As of today (12-19-2023), the arXiv submission of the original deep Q-learning approach to achieve superhuman performance on ATARI games has turned a decade old. The original approach, ...
DeepQZero's user avatar
  • 1,703
0 votes
1 answer
86 views

How do I update Q-values in Q-learning when rewards may only be received after many actions?

I am working on a Q-learning system where the agent may well (and almost always) have to take many actions before a reward can be given to the agent (or more so, the notion of a reward in my context ...
Hera Sutton's user avatar
1 vote
1 answer
1k views

When to use Pruning, Quantization , Distillation and others when optimizing speed

I want to understand how to optimize models for inference speed and am seeking some advice and best practices for the same. I am a little bit aware of the concepts of pruning, quantization, and ...
Hiren Namera's user avatar
4 votes
2 answers
3k views

What are the differences between seq2seq and encoder-decoder architectures?

I've read many tutorials online that use both words interchangeably. When I search and find that they are the same, why not just use one word since they have the same definition?
user avatar
6 votes
2 answers
614 views

Does increasing the number of Q functions in Q-Learning scale?

Q-Learning (Watkins, 1989) uses a single function to estimate the value of actions and to choose the next action. Double Q-Learning (Hasselt, 2010) extends this and uses two functions which are ...
foreverska's user avatar
  • 1,559
2 votes
2 answers
141 views

Can we solve the environment with only the linear and angular position through Q-Learning?

I'm trying to solve the cartpole-v1 gym environment with only the linear and angular position, but the mean reward of the last 100 episodes isn't greater than 20 rewards. The longest train i made was ...
Vitor Martins's user avatar
1 vote
0 answers
36 views

Why slow-changing policy invalidates Double DQN approach in TD3 paper?

In the paper describing TD3 (https://arxiv.org/abs/1802.09477), the authors say that they could not effectively address the Q-learning overestimation bias by using different networks for maximizing ...
Jerry Ding's user avatar
1 vote
1 answer
133 views

Why are these two implementations of the $\epsilon$-greedy policy different?

According to the book Reinforcement Learning An Introduction, the epsilon greedy policy can generally implemented as: $$ \pi(a|s) = \begin{cases} \frac{\epsilon}{|A|} + 1 - \epsilon & \text{if } ...
kklaw's user avatar
  • 195
2 votes
1 answer
541 views

What are the similarities between Q-learning and Value Iteration?

This is the explanation of value iteration in our notes where you keep applying bellman optimality equation till it stops changing and then acting greedily wrt the value function gives the optimal ...
ace239's user avatar
  • 23
1 vote
1 answer
64 views

Can Q-learning be used to create new creative solutions by combining different factors and characteristics?

References from Wikipedia: https://en.wikipedia.org/wiki/Q-learning https://en.wikipedia.org/wiki/Markov_decision_process Q-learning can be used to create new creative solutions, combining different ...
will The J's user avatar
1 vote
1 answer
56 views

In Q-learning, states need to be just X and Y positions of the agent, or a state can be several other characteristics?

For example, in this article: https://www.learndatasci.com/tutorials/reinforcement-q-learning-scratch-python-openai-gym/, which explains Q-learnig, teaches the Smartcab problem, the environment is a ...
will The J's user avatar
2 votes
1 answer
60 views

Is Q-learning limited to just visual scenarios, or is it much broader and can it be used to solve non-visual problems as well?

For example, in this article: https://www.learndatasci.com/tutorials/reinforcement-q-learning-scratch-python-openai-gym/, which explains Q-learnig, teaches the Smartcab problem, it has a visual ...
will The J's user avatar
1 vote
2 answers
313 views

In Q-learning, Am I the one who will define the way in which actions allow the agent to interact with the environment? And the interactions will vary?

In Q-learning, am I the one who will define the way in which actions allow the agent to interact with the environment, so that the way in which actions allow the agent to interact with the environment ...
will The J's user avatar
1 vote
1 answer
212 views

Could someone give a very simple example of Q-learning in a very small environment? [closed]

I would really like to see an example of Q-learning that I could read, so that I can learn Q-learning from scratch. I read some articles on the internet, but I found it a little difficult to ...
will The J's user avatar

1
2 3 4 5
17