Questions tagged [convergence]

For questions related to the convergence of AI algorithms.

Filter by
Sorted by
Tagged with
0 votes
0 answers
7 views

A practical example of uniform convergence of a hypothesis class

From definition of uniform convergence: We say that a hypothesis class $\mathcal{H}$ has the uniform convergence property (w.r.t. a domain Z and a loss function $l$) if there exists a function $m_{H}^...
Tran Khanh's user avatar
1 vote
1 answer
31 views

Applicability of Holland's Schema Theorem to Genetic Algorithms with Non-Binary Individual Representations

I'm currently working on a problem formulation that requires non-binary individual representations in a genetic algorithm (GA). I've been exploring Holland's Schema Theorem as a theoretical basis for ...
CES's user avatar
  • 11
1 vote
2 answers
127 views

What is the impact of the initialization of weights in the performance of a neural network in machine learning?

In my own experience, weight initialization matters for model convergence. Theoretically, can different weight initialization methods eventually converge to the same optimal solution? Are their ...
Robin van Hoorn's user avatar
0 votes
1 answer
45 views

How big is the threshold that is usually used in determining the convergence of loss values in deep learning?

In deep learning, one way to determine whether the training has converged is to observe the movement of the loss values over iterations or epochs. One can choose any $\epsilon$ threshold and any ...
poglhar's user avatar
  • 23
5 votes
2 answers
2k views

What is curriculum learning in reinforcement learning?

I recently came across the term "curriculum learning" in the context of DRL and was intrigued by its potential to improve the learning process. As such, what is curriculum learning? And how ...
Robin van Hoorn's user avatar
0 votes
0 answers
19 views

Model starts off stronger than SOTA, but doesn't maintain advantage

In general, what could cause a Neural Network to start with better results than a similar model, but then lose it's advantage after a few epochs ? If a NN starts off about 10% better than a similar ...
Liam F-A's user avatar
2 votes
0 answers
88 views

Can Q-learning and other RL algorithms solve CNF SAT?

I encountered a question about solving CNF SAT using reinforcement learning: A state is a partial substitution to the variables, and each action is choosing an empty variable and set its value (to <...
Dani's user avatar
  • 121
0 votes
0 answers
28 views

Effectiveness of DNN training with reduced Batch randomness

So here's an example set to help explain my doubt. Suppose I have 80,000 total images available for a DNN training task. With a batch size of 32, that is 2500 batches. Now let's say I partition the ...
Keshav Vinayak Jha's user avatar
0 votes
0 answers
238 views

Convergence of Value Iteration for Discount factor of 1

Given this pseudo code for value iteration: In the case of gamma=1, under what conditions on the MDP will we still be able to find the optimal policy?
Toffe1369's user avatar
0 votes
0 answers
23 views

Are there are any truly chaotic systems in deep learning where convergence is not possible?

In the cases I have seen, neural networks have always converged (i.e., the loss function for the training data asymptotically approached some constant value). I have seen loss functions oscillate, ...
Snehal Patel's user avatar
1 vote
1 answer
527 views

Why is policy iteration guaranteed to converge to the global optimum? [duplicate]

In reinforcement learning, what guarantees that policy iteration would find the globally optimal solution and not just any local maximum? I'm reading the book "Reinforcement Learning: An ...
martinkunev's user avatar
3 votes
1 answer
125 views

Prooving the convergence rate of estimators (machine Learning)

I want to estimate a quantity and have two choices for estimators (they both sample from the same distribution). I suspect one of them has a higher variance and thus a slower convergence rate. I want ...
postnubilaphoebus's user avatar
1 vote
0 answers
66 views

Training a neural network simultaneously with two different loss functions rather than considering the weighted sum

This is a follow up on the already asked question: Is the neural network 100% accurate on training data if epoch loss is minimized to 0? I want to train a neural network that works as an approximator ...
Acad's user avatar
  • 111
3 votes
1 answer
127 views

Learning an identity function with convolutional networks

I am trying to train networks to achieve what I expected to be a trivial task: learn the identity mapping. However, this is very hard to achieve, and the optimization is hard. Moreover, I don't want ...
Franco Marchesoni's user avatar
1 vote
2 answers
932 views

Should PPO always converge toward the global optimum?

I'm trying to "solve" the OpenAI gym environment "Humanoid-v3" using PPO. I got it to work to some degree (The NN is learning a policy and perfecting it. Average reward of about 5....
pjungk's user avatar
  • 13
3 votes
2 answers
217 views

Is there any variant of perceptron convergence algorithm that ensures uniqueness?

The perceptron convergence algorithm given below ensures the convergence of weights of the perceptron provided enough data points and iterations. Although it ensures convergence by finally getting a ...
hanugm's user avatar
  • 3,612
2 votes
1 answer
242 views

Is it possible learning convergence is lost in Reinforcement Learning as the state space grows?

I am new in the AI field and I am trying to use Reinforcement Learning. Specifically, I am using tabular Q-Learning and SARSA algorithms to solve a sequential decision making problem. (I am using <...
Aquila's user avatar
  • 33
2 votes
0 answers
441 views

Do learning rate schedulers conflict with or prevent convergence of the Adam optimiser?

An article on https://spell.ml says Because Adam manages learning rates internally, it's incompatible with most learning rate schedulers. Anything more complicated than simple learning warmup and/or ...
Jack G's user avatar
  • 21
1 vote
2 answers
2k views

Why and how can the policy and value iteration methods converge to the OPTIMAL point?

I am reading Reinforcement Learning: An Introduction by Sutton & Barto. According to this textbook, as far as I understood, the authors claim that the policy and value iteration methods converge ...
Danny_Kim's user avatar
  • 113
0 votes
3 answers
162 views

What can I infer if my model is converging extremely fast?

I am running a model with fixed hyperparameters. To my surprise/shock, the model converged extremely fast with the least loss possible. I want to know the causes of this phenomenon. I have the ...
hanugm's user avatar
  • 3,612
3 votes
0 answers
54 views

How to fix high variance of the returns on a 2d env?

I'm trying to train an agent on a self-written 2d env, and it just doesn't converge to the solution. It is basically a 2d game where you have to move a small circle around the screen and try to avoid ...
debrises's user avatar
2 votes
0 answers
166 views

Why does the number of input tokens to an LSTM have an impact on the convergence of Integrated Gradients?

Background I am computing the attribution scores for a simple LSTM model using Integrated Gradients. This method defines the contribution of a feature to a model prediction by integrating over the ...
jumelet's user avatar
  • 71
1 vote
1 answer
325 views

Is there a tutorial for understanding the proof of convergence for TD learning?

I'm reading the article An Analysis of Temporal-Difference Learning with Function Approximation (1997), but the mathematics inside seems overly complicated for me. Answers to some similar questions ...
heiyezhu6324's user avatar
1 vote
0 answers
106 views

Hand Landmark Detector Not Converging

I'm currently trying to train a custom model with TensorFlow to detect 17 landmarks/keypoints on each of 2 hands shown in an image (fingertips, first knuckles, bottom knuckles, wrist, and palm), for ...
Sam Skinner's user avatar
2 votes
2 answers
711 views

How does $\alpha$ affect the convergence of the TD algorithm?

In Temporal-Difference Learning, we update our value function by $V\left(S_{t}\right) \leftarrow V\left(S_{t}\right)+\alpha\left(R_{t+1}+\gamma V\left(S_{t+1}\right)-V\left(S_{t}\right)\right)$ If we ...
XXX's user avatar
  • 143
2 votes
0 answers
212 views

Why does TD (0) converge to the MLE solution of the Markov model?

Why does TD (0) converge to the MLE solution of the Markov model? Let's take the Example 6.4 in Sutton and Barto's book as an example. Example 6.4: You are the Predictor Place yourself now in the ...
XXX's user avatar
  • 143
1 vote
0 answers
12 views

References for the convergence of gradient-based algorithms for training neural networks

I'm looking for some good references that give convergence results of training neural networks. I'm decently familiar with works that analyze the convergence of SGD, and, in particular, I really like ...
Taw's user avatar
  • 1,201
5 votes
2 answers
5k views

How to check whether my loss function is convex or not?

Loss functions are useful in calculating loss and then we can update the weights of a neural network. The loss function is thus useful in training neural networks. Consider the following excerpt from ...
hanugm's user avatar
  • 3,612
1 vote
0 answers
45 views

Is there any work that applies the approach in "Finite-Sample Convergence Rates for Q-Learning and Indirect Algorithms" to standard Q-learning?

I am trying to mathematically characterize the finite sample convergence rates for Q-learning. To this end, I have read the following papers Learning rates for Q-learning, by Eyal Even-Dar et al.; ...
Jose Maria Gutierrez's user avatar
0 votes
0 answers
63 views

Why don't we use this intialization with SGD rather than random?

Suppose I have a loss function as a polynomial with its variables being the weights of a network I wish to tune. Now, we want to find the minima of the loss function - so basically ...
neel g's user avatar
  • 136
4 votes
1 answer
303 views

Does elitism cause premature convergence in genetic algorithms?

I have a genetic algorithm which is working fairly well. It's got all the standard operators, including initial random population, crossover ratio, mutation rate, degree of mutation, etc. This works ...
Pittsburgh DBA's user avatar
5 votes
1 answer
904 views

Does the policy iteration convergence hold for finite-horizon MDP?

Most RL books (Sutton & Barto, Bertsekas, etc.) talk about policy iteration for infinite-horizon MDPs. Does the policy iteration convergence hold for finite-horizon MDP? If yes, how can we derive ...
user529295's user avatar
5 votes
1 answer
746 views

Why does Q-learning converge under 100% exploration rate?

I am working on this assignment where I made the agent learn state-action values (Q-values) with Q-learning and 100% exploration rate. The environment is the classic gridworld as shown in the ...
Rim Sleimi's user avatar
2 votes
1 answer
106 views

Why do I get the best policy before Q values converge using DQN?

I have implemented DQN algorithm and wonder why during testing, the best performance is achieved by a policy from about 300 episode, when mean Q values converge at about 800 episode? Mean Q-values ...
user avatar
1 vote
1 answer
287 views

If $\alpha$ decreases over time, why is Q-learning guaranteed to converge?

Q-Learning is guaranteed to converge if $\alpha$ decreases over time. On page 161 of the RL book by Sutton and Barto, 2nd edition, section 8.1, they write that Dyna-Q is guaranteed to converge if each ...
user8714896's user avatar
1 vote
0 answers
25 views

React on train-validation curve after trening

I have a regression task that I tray to solve with AI. I have around 6M rows with about 30 columns. (originally there was 100, but I reduce it with drop feature importance) I understand basic ...
Marko Zadravec's user avatar
1 vote
0 answers
77 views

How to have closer validation loss and training loss in training a CNN

I am using an AlexNet architecture as my Convolutional Neural Network. A learning rate of 0.00007 and 128 batch_size. I have 20000 data and 10% test, 40% validation, and 50% for training. I used 100 ...
SahaTib's user avatar
  • 150
3 votes
1 answer
2k views

How to determine if Q-learning has converged in practice?

I am using Q-learning and SARSA to solve a problem. The agent learns to go from the start to the goal without falling in the holes. At each state, I can choose the action corresponding to the maximum ...
WANGWANGZI's user avatar
1 vote
0 answers
234 views

Is Monte Carlo tree search guaranteed to converge to the optimal solution in two player zero-sum stochastic games?

I'm aware that convergence proofs for Monte Carlo tree search exist in the case of deterministic zero sum games and Markov decision processes. I have come across research which applies MCTS to zero-...
markr3656's user avatar
2 votes
1 answer
117 views

How much can an inclusion of the number of iterations have on the training of an MLP?

My doubt is like this : Suppose we have an MLP. In an MLP, as per the backprop algorithm (back-propagation algorithm), the correction applied to each weight is : $$ w_{ij} := -\eta\frac{\partial E}{\...
Spectre's user avatar
  • 177
4 votes
1 answer
1k views

When do SARSA and Q-Learning converge to optimal Q values?

Here's another interesting multiple-choice question that puzzles me a bit. In tabular MDPs, if using a decision policy that visits all states an infinite number of times, and in each state, randomly ...
stoic-santiago's user avatar
2 votes
0 answers
197 views

How can deep Q-learning converge if the targets may not be correct?

In deep Q-learning, $Q(s, a)$ and $Q'(s, a)$ are predicted or estimated by the neural network itself. In supervised learning, the target value is a true unbiased value. However, this isn't the case in ...
Chukwudi's user avatar
  • 359
2 votes
0 answers
77 views

If the performance of an RL agent in a partially observable environment is "good", is this likely only accidental?

In my research, I remember to have read that, in case of an environment which can be modeled by partially observable MDP, there are no convergence guarantees (unfortunately, I do not find the paper ...
unter_983's user avatar
  • 331
8 votes
0 answers
263 views

Is the Bellman equation that uses sampling weighted by the Q values (instead of max) a contraction?

It is proved that the Bellman update is a contraction (1). Here is the Bellman update that is used for Q-Learning: $$Q_{t+1}(s, a) = Q_{t}(s, a) + \alpha*(r(s, a, s') + \gamma \max_{a^*} (Q_{t}(s', ...
sirfroggy's user avatar
2 votes
0 answers
947 views

Why is DDPG not learning and it does not converge?

I have used a different setting, but DDPG is not learning and it does not converge. I have used these codes 1,2, and 3 and I used different optimizers, activation functions, and learning rate but ...
I_Al-thamary's user avatar
2 votes
1 answer
121 views

What is convergence analysis, and why is it needed in reinforcement learning?

While reading a paper about Q-learning in network energy consumption, I came across the section on convergence analysis. Does anyone know what convergence analysis is, and why is convergence analysis ...
Daniel Koh's user avatar
2 votes
0 answers
272 views

If the minimum Q value is decreasing and the maximum Q value increasing, is this a sign that dueling double DQN is diverging?

I'm training a dueling double DQN agent with prioritized replay buffer and notice that the min Q values are decreasing, while the max Q values are increasing. Is this a sign that it is diverging? ...
Athena Wisdom's user avatar
3 votes
1 answer
873 views

If deep Q-learning starts to choose only one action, is this a sign that the algorithm diverged?

I'm working on a deep q-learning model in an infinite horizon problem, with a continous state space and 3 possible actions. I'm using a neural network to approximate the action-value function. ...
unter_983's user avatar
  • 331
3 votes
0 answers
119 views

Convergence of a delayed policy update Q-learning

I thought about an algorithm that twists the standard Q-learning slightly, but I am not sure whether convergence to the optimal Q-value could be guaranteed. The algorithm starts with an initial ...
Lyapunov1729's user avatar
6 votes
1 answer
2k views

What are the conditions of convergence of temporal-difference learning?

In reinforcement learning, temporal difference seem to update the value function in each new iteration of experience absorbed from the environment. What would be the conditions for temporal-...
MJeremy's user avatar
  • 163