Skip to main content

Questions tagged [convergence]

For questions related to the convergence of AI algorithms.

33 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
8 votes
0 answers
275 views

Is the Bellman equation that uses sampling weighted by the Q values (instead of max) a contraction?

It is proved that the Bellman update is a contraction (1). Here is the Bellman update that is used for Q-Learning: $$Q_{t+1}(s, a) = Q_{t}(s, a) + \alpha*(r(s, a, s') + \gamma \max_{a^*} (Q_{t}(s', ...
sirfroggy's user avatar
4 votes
0 answers
2k views

Why is it hard to prove the convergence of the deep Q-learning algorithm?

Why is it hard to prove the convergence of the DQN algorithm? We know that the tabular Q-learning algorithm converges to the optimal Q-values, and with a linear approximator convergence is proved. ...
Afshin Oroojlooy's user avatar
4 votes
1 answer
215 views

Why is my implementation of REINFORCE algorithm for portfolio optimization not converging?

I'm trying to implement the Reinforce algorithm (Monte Carlo policy gradient) in order to optimize a portfolio of 94 stocks on a daily basis (I have suitable historical data to achieve this). The idea ...
BGa's user avatar
  • 229
3 votes
0 answers
92 views

Is improving a Neural Network really just "trial and error"?

After asking on StackOverflow, I was redirected here, so I'm reposting this question. I am a PhD student in Computational Physics and I've started to study a bit of Neural Networks, and decided to try ...
Mauro Giliberti's user avatar
3 votes
2 answers
255 views

Is there any variant of perceptron convergence algorithm that ensures uniqueness?

The perceptron convergence algorithm given below ensures the convergence of weights of the perceptron provided enough data points and iterations. Although it ensures convergence by finally getting a ...
hanugm's user avatar
  • 3,870
3 votes
0 answers
59 views

How to fix high variance of the returns on a 2d env?

I'm trying to train an agent on a self-written 2d env, and it just doesn't converge to the solution. It is basically a 2d game where you have to move a small circle around the screen and try to avoid ...
debrises's user avatar
3 votes
0 answers
134 views

Convergence of a delayed policy update Q-learning

I thought about an algorithm that twists the standard Q-learning slightly, but I am not sure whether convergence to the optimal Q-value could be guaranteed. The algorithm starts with an initial ...
Lyapunov1729's user avatar
2 votes
0 answers
103 views

Can Q-learning and other RL algorithms solve CNF SAT?

I encountered a question about solving CNF SAT using reinforcement learning: A state is a partial substitution to the variables, and each action is choosing an empty variable and set its value (to <...
Dani's user avatar
  • 121
2 votes
0 answers
511 views

Do learning rate schedulers conflict with or prevent convergence of the Adam optimiser?

An article on https://spell.ml says Because Adam manages learning rates internally, it's incompatible with most learning rate schedulers. Anything more complicated than simple learning warmup and/or ...
Jack G's user avatar
  • 21
2 votes
0 answers
176 views

Why does the number of input tokens to an LSTM have an impact on the convergence of Integrated Gradients?

Background I am computing the attribution scores for a simple LSTM model using Integrated Gradients. This method defines the contribution of a feature to a model prediction by integrating over the ...
jumelet's user avatar
  • 71
2 votes
0 answers
271 views

Why does TD (0) converge to the MLE solution of the Markov model?

Why does TD (0) converge to the MLE solution of the Markov model? Let's take the Example 6.4 in Sutton and Barto's book as an example. Example 6.4: You are the Predictor Place yourself now in the ...
XXX's user avatar
  • 143
2 votes
0 answers
205 views

How can deep Q-learning converge if the targets may not be correct?

In deep Q-learning, $Q(s, a)$ and $Q'(s, a)$ are predicted or estimated by the neural network itself. In supervised learning, the target value is a true unbiased value. However, this isn't the case in ...
Chukwudi's user avatar
  • 369
2 votes
0 answers
79 views

If the performance of an RL agent in a partially observable environment is "good", is this likely only accidental?

In my research, I remember to have read that, in case of an environment which can be modeled by partially observable MDP, there are no convergence guarantees (unfortunately, I do not find the paper ...
unter_983's user avatar
  • 331
2 votes
0 answers
1k views

Why is DDPG not learning and it does not converge?

I have used a different setting, but DDPG is not learning and it does not converge. I have used these codes 1,2, and 3 and I used different optimizers, activation functions, and learning rate but ...
I_Al-thamary's user avatar
2 votes
0 answers
292 views

If the minimum Q value is decreasing and the maximum Q value increasing, is this a sign that dueling double DQN is diverging?

I'm training a dueling double DQN agent with prioritized replay buffer and notice that the min Q values are decreasing, while the max Q values are increasing. Is this a sign that it is diverging? ...
Athena Wisdom's user avatar
2 votes
0 answers
63 views

How to show Monte Carlo methods converge to an estimate which minimizes mean squared error?

In chapter six of Sutton and Barto (p.128), they claim Monte Carlo methods converge to an estimate minimizing the mean squared error. How can this be shown formally? Bump
fool's user avatar
  • 203
2 votes
0 answers
49 views

How to know when a Environment will yield a deterministic model

Given enough experiment data on time taken for objects to fall to earth from different heights, one can create various models that will accurately predict the time it will take for an object falling ...
benbyford's user avatar
  • 348
2 votes
0 answers
158 views

Are there reinforcement learning algorithms that ensure convergence for continuous state space problems?

The Q-learning does not guarantee convergence for continuous state space problems (Why doesn't Q-learning converge when using function approximation?). In that case, is there an algorithm which ...
shunyo's user avatar
  • 133
1 vote
0 answers
75 views

Training a neural network simultaneously with two different loss functions rather than considering the weighted sum

This is a follow up on the already asked question: Is the neural network 100% accurate on training data if epoch loss is minimized to 0? I want to train a neural network that works as an approximator ...
Acad's user avatar
  • 111
1 vote
0 answers
114 views

Hand Landmark Detector Not Converging

I'm currently trying to train a custom model with TensorFlow to detect 17 landmarks/keypoints on each of 2 hands shown in an image (fingertips, first knuckles, bottom knuckles, wrist, and palm), for ...
Sam Skinner's user avatar
1 vote
0 answers
12 views

References for the convergence of gradient-based algorithms for training neural networks

I'm looking for some good references that give convergence results of training neural networks. I'm decently familiar with works that analyze the convergence of SGD, and, in particular, I really like ...
Taw's user avatar
  • 1,251
1 vote
0 answers
46 views

Is there any work that applies the approach in "Finite-Sample Convergence Rates for Q-Learning and Indirect Algorithms" to standard Q-learning?

I am trying to mathematically characterize the finite sample convergence rates for Q-learning. To this end, I have read the following papers Learning rates for Q-learning, by Eyal Even-Dar et al.; ...
Jose Maria Gutierrez's user avatar
1 vote
0 answers
27 views

React on train-validation curve after trening

I have a regression task that I tray to solve with AI. I have around 6M rows with about 30 columns. (originally there was 100, but I reduce it with drop feature importance) I understand basic ...
Marko Zadravec's user avatar
1 vote
0 answers
84 views

How to have closer validation loss and training loss in training a CNN

I am using an AlexNet architecture as my Convolutional Neural Network. A learning rate of 0.00007 and 128 batch_size. I have 20000 data and 10% test, 40% validation, and 50% for training. I used 100 ...
SahaTib's user avatar
  • 150
1 vote
0 answers
263 views

Is Monte Carlo tree search guaranteed to converge to the optimal solution in two player zero-sum stochastic games?

I'm aware that convergence proofs for Monte Carlo tree search exist in the case of deterministic zero sum games and Markov decision processes. I have come across research which applies MCTS to zero-...
markr3656's user avatar
1 vote
0 answers
132 views

When does Monte Carlo linear function approximation converge?

In this Stanford lecture (minute 35:47 and 37:00), the professor says that Monte Carlo (MC) linear function approximation does not always converge, and she gives an example. In general, when does MC ...
dato nefaridze's user avatar
1 vote
0 answers
123 views

Does SARSA(0) converge to the optimal policy in expectation if the Robbins-Monro conditions are removed?

The conditions of convergence of SARSA(0) to the optimal policy are : The Robbins-Monro conditions above hold for $α_t$. Every state-action pair is visited infinitely often The policy is greedy with ...
KaneM's user avatar
  • 309
1 vote
0 answers
34 views

Imposing contraints on sequence of image classifications

Are there example implementations of networks that apply constraints across sequences of image classifications where class labels are ordinal numbers? For example, to cause the output of a CNN to ...
bucklera's user avatar
0 votes
0 answers
10 views

A practical example of uniform convergence of a hypothesis class

From definition of uniform convergence: We say that a hypothesis class $\mathcal{H}$ has the uniform convergence property (w.r.t. a domain Z and a loss function $l$) if there exists a function $m_{H}^...
Tran Khanh's user avatar
0 votes
1 answer
80 views

How big is the threshold that is usually used in determining the convergence of loss values in deep learning?

In deep learning, one way to determine whether the training has converged is to observe the movement of the loss values over iterations or epochs. One can choose any $\epsilon$ threshold and any ...
poglhar's user avatar
  • 23
0 votes
0 answers
19 views

Model starts off stronger than SOTA, but doesn't maintain advantage

In general, what could cause a Neural Network to start with better results than a similar model, but then lose it's advantage after a few epochs ? If a NN starts off about 10% better than a similar ...
Liam F-A's user avatar
0 votes
0 answers
372 views

Convergence of Value Iteration for Discount factor of 1

Given this pseudo code for value iteration: In the case of gamma=1, under what conditions on the MDP will we still be able to find the optimal policy?
Toffe1369's user avatar
0 votes
0 answers
71 views

Why don't we use this intialization with SGD rather than random?

Suppose I have a loss function as a polynomial with its variables being the weights of a network I wish to tune. Now, we want to find the minima of the loss function - so basically ...
neel g's user avatar
  • 146