Questions tagged [convergence]

For questions related to the convergence of AI algorithms.

Filter by
Sorted by
Tagged with
1
vote
0answers
10 views

How to check whether my loss function is convex or not?

Loss functions are useful in calculating loss and then we can update weights of a neural network. Loss function is thus useful in training neural networks. Consider the following excerpt from this ...
0
votes
0answers
15 views

Is input standardisation better than input normalisation?

Consider a network which takes samples of single values. And consider the training set of 5 samples: $$ inp = [5, 6, 7, 8, 9] $$ Input normalisation: $$ min = 5, max = 9, span = 9-5 = 4 \\ Input1 = [(...
0
votes
0answers
13 views

MDP - policy iteration convergence proof

I'm currently taking an Intro to AI course, and we've learned about MDP's and specifically about policy iteration. When we talked about the convergence of the policy iteration, it was mentioned that ...
1
vote
0answers
21 views

Is there any work that applies the approach in “Finite-Sample Convergence Rates for Q-Learning and Indirect Algorithms” to standard Q-learning?

I am trying to mathematically characterize the finite sample convergence rates for Q-learning. To this end, I have read the following papers Learning rates for Q-learning, by Eyal Even-Dar et al.; ...
0
votes
0answers
35 views

Why don't we use this intialization with SGD rather than random?

Suppose I have a loss function as a polynomial with its variables being the weights of a network I wish to tune. Now, we want to find the minima of the loss function - so basically ...
4
votes
1answer
37 views

Does elitism cause premature convergence in genetic algorithms?

I have a genetic algorithm which is working fairly well. It's got all the standard operators, including initial random population, crossover ratio, mutation rate, degree of mutation, etc. This works ...
5
votes
1answer
205 views

Does the policy iteration convergence hold for finite-horizon MDP?

Most RL books (Sutton & Barto, Bertsekas, etc.) talk about policy iteration for infinite-horizon MDPs. Does the policy iteration convergence hold for finite-horizon MDP? If yes, how can we derive ...
5
votes
1answer
190 views

Why does Q-learning converge under 100% exploration rate?

I am working on this assignment where I made the agent learn state-action values (Q-values) with Q-learning and 100% exploration rate. The environment is the classic gridworld as shown in the ...
2
votes
1answer
68 views

Why do I get the best policy before Q values converge using DQN?

I have implemented DQN algorithm and wonder why during testing, the best performance is achieved by a policy from about 300 episode, when mean Q values converge at about 800 episode? Mean Q-values ...
1
vote
1answer
69 views

If $\alpha$ decreases over time, why is Q-learning guaranteed to converge?

Q-Learning is guaranteed to converge if $\alpha$ decreases over time. On page 161 of the RL book by Sutton and Barto, 2nd edition, section 8.1, they write that Dyna-Q is guaranteed to converge if each ...
1
vote
0answers
18 views

React on train-validation curve after trening

I have a regression task that I tray to solve with AI. I have around 6M rows with about 30 columns. (originally there was 100, but I reduce it with drop feature importance) I understand basic ...
1
vote
0answers
41 views

How to have closer validation loss and training loss in training a CNN

I am using an AlexNet architecture as my Convolutional Neural Network. A learning rate of 0.00007 and 128 batch_size. I have 20000 data and 10% test, 40% validation, and 50% for training. I used 100 ...
3
votes
1answer
302 views

How to determine if Q-learning has converged in practice?

I am using Q-learning and SARSA to solve a problem. The agent learns to go from the start to the goal without falling in the holes. At each state, I can choose the action corresponding to the maximum ...
1
vote
0answers
56 views

Is Monte Carlo tree search guaranteed to converge to the optimal solution in two player zero-sum stochastic games?

I'm aware that convergence proofs for Monte Carlo tree search exist in the case of deterministic zero sum games and Markov decision processes. I have come across research which applies MCTS to zero-...
2
votes
1answer
30 views

How much can an inclusion of the number of iterations have on the training of an MLP?

My doubt is like this : Suppose we have an MLP. In an MLP, as per the backprop algorithm (back-propagation algorithm), the correction applied to each weight is : $$ w_{ij} := -\eta\frac{\partial E}{\...
4
votes
1answer
279 views

When do SARSA and Q-Learning converge to optimal Q values?

Here's another interesting multiple-choice question that puzzles me a bit. In tabular MDPs, if using a decision policy that visits all states an infinite number of times, and in each state, randomly ...
1
vote
0answers
32 views

How can deep Q-learning converge if the targets may not be correct?

In deep Q-learning, $Q(s, a)$ and $Q'(s, a)$ are predicted or estimated by the neural network itself. In supervised learning, the target value is a true unbiased value. However, this isn't the case in ...
2
votes
0answers
56 views

If the performance of an RL agent in a partially observable environment is “good”, is this likely only accidental?

In my research, I remember to have read that, in case of an environment which can be modeled by partially observable MDP, there are no convergence guarantees (unfortunately, I do not find the paper ...
6
votes
0answers
165 views

Is the Bellman equation that uses sampling weighted by the Q values (instead of max) a contraction?

It is proved that the Bellman update is a contraction (1). Here is the Bellman update that is used for Q-Learning: $$Q_{t+1}(s, a) = Q_{t}(s, a) + \alpha*(r(s, a, s') + \gamma \max_{a^*} (Q_{t}(s', ...
2
votes
0answers
388 views

Why is DDPG not learning and it does not converge?

I have used a different setting, but DDPG is not learning and it does not converge. I have used these codes 1,2, and 3 and I used different optimizers, activation functions, and learning rate but ...
2
votes
1answer
46 views

What is convergence analysis, and why is it needed in reinforcement learning?

While reading a paper about Q-learning in network energy consumption, I came across the section on convergence analysis. Does anyone know what convergence analysis is, and why is convergence analysis ...
2
votes
0answers
100 views

If the minimum Q value is decreasing and the maximum Q value increasing, is this a sign that dueling double DQN is diverging?

I'm training a dueling double DQN agent with prioritized replay buffer and notice that the min Q values are decreasing, while the max Q values are increasing. Is this a sign that it is diverging? ...
1
vote
1answer
85 views

If deep Q-learning starts to choose only one action, is this a sign that the algorithm diverged?

I'm working on a deep q-learning model in an infinite horizon problem, with a continous state space and 3 possible actions. I'm using a neural network to approximate the action-value function. ...
3
votes
0answers
62 views

Convergence of a delayed policy update Q-learning

I thought about an algorithm that twists the standard Q-learning slightly, but I am not sure whether convergence to the optimal Q-value could be guaranteed. The algorithm starts with an initial ...
5
votes
1answer
421 views

What are the conditions of convergence of temporal-difference learning?

In reinforcement learning, temporal difference seem to update the value function in each new iteration of experience absorbed from the environment. What would be the conditions for temporal-...
2
votes
1answer
605 views

Why isn't my implementation of A2C for the the atari pong game converging?

I have two different implementations with PyTorch of the Atari Pong game using A2C algorithm. Both implementations are similar, but some portion are different. https://colab.research.google.com/drive/...
2
votes
0answers
308 views

Why is it hard to prove the convergence of the deep Q-learning algorithm?

Why is it hard to prove the convergence of the DQN algorithm? We know that the tabular Q-learning algorithm converges to the optimal Q-values, and with a linear approximator convergence is proved. ...
3
votes
1answer
218 views

Why do RL implementations converge on one action?

I have seen this happening in implementations of state-of-the-art RL algorithms where the model converges to a single action over time after multiple training iterations. Are there some general ...
1
vote
0answers
66 views

When does Monte Carlo linear function approximation converge?

In this Stanford lecture (minute 35:47 and 37:00), the professor says that Monte Carlo (MC) linear function approximation does not always converge, and she gives an example. In general, when does MC ...
0
votes
1answer
63 views

Is it a good idea to overfit on a small part of your data for faster model convergence?

I working on a classification problem that needs to detect patterns on a time serie. Basically, there's a catch-all class that means "no pattern detected", the other are for the specific patterns. The ...
3
votes
1answer
178 views

Neural network doesn't seem to converge with ReLU but it does with Sigmoid?

I'm not really sure if this is the sort of question to ask on here, since it is less of a general question about AI and more about the coding of it, however I thought it wouldn't fit on stack overflow....
3
votes
3answers
264 views

Why can the Bellman equation be turned into an update rule?

In chapter 4.1 of Sutton's book, the Bellman equation is turned into an update rule by simply changing the indices of it. How is it mathematically justified? I didn't quite get the initiation of why ...
1
vote
1answer
105 views

Why isn't the implementation of my policy evaluation for a simple MDP converging?

I am trying to code out a policy evaluation algorithm to find the $V^\pi(s)$ for all states. The following diagram below shows the MDP. In this case i let p = q = 0.5. the rewards for each states ...
2
votes
1answer
380 views

Is there an advantage in decaying $\epsilon$ during Q-Learning?

If the agent is following an $\epsilon$-greedy policy derived from Q, is there any advantage to decaying $\epsilon$ even though $\epsilon$ decay is not required for convergence ?
1
vote
0answers
71 views

Does SARSA(0) converge to the optimal policy in expectation if the Robbins-Monro conditions are removed?

The conditions of convergence of SARSA(0) to the optimal policy are : The Robbins-Monro conditions above hold for $α_t$. Every state-action pair is visited infinitely often The policy is greedy with ...
1
vote
2answers
253 views

What are the conditions for the convergence of SARSA to the optimal value function?

Is it correct that for SARSA to converge to the optimal value function (and policy) The learning rate parameter $\alpha$ must satisfy the conditions: $$\sum \alpha_{n^k(s,a)} =\infty \quad \text{and}...
0
votes
1answer
34 views

Is it possible to use deeplearning with spark (with a distributed databases as HDFS or Cassandra)?

If it is possible, will it be really useful or the model will end up converging very early(with a typical optimum learning rate) ? Any content on this topic will be helpful for me.
1
vote
1answer
157 views

Does TD(0) prediction require Robbins-Monro conditions to converge to the value function?

Does the learning rate parameter $\alpha$ require the Robbins-Monro conditions below for the TD(0) algorithm to converge to the true value function of a policy? $$\sum \alpha_t =\infty \quad \text{...
2
votes
1answer
703 views

Is there a simple proof of the convergence of TD(0)?

Does anybody know a simple proof of the convergence of the TD(0) value function prediction algorithm?
3
votes
1answer
293 views

Why does the error of my LSTM not decrease after 10 epochs?

Despite the problem being very simple, I was wondering why an LSTM network was not able to converge to a decent solution. ...
7
votes
1answer
157 views

Why does reinforcement learning using a non-linear function approximator diverge when using strongly correlated data as input?

While reading the DQN paper, I found that randomly selecting and learning samples reduced divergence in RL using a non-linear function approximator (e.g a neural network). So, why does Reinforcement ...
1
vote
0answers
29 views

Imposing contraints on sequence of image classifications

Are there example implementations of networks that apply constraints across sequences of image classifications where class labels are ordinal numbers? For example, to cause the output of a CNN to ...
4
votes
1answer
239 views

When exactly is a model considered over-parameterized?

When exactly is a model considered over-parameterized? There are some recent researches in Deep Learning about the role of over-parameterization toward generalization, so it would be nice if I can ...
6
votes
2answers
6k views

What is convergence in machine learning?

I came across this answer on Quora, but it was pretty sparse. I'm looking for specific meanings in the context of machine learning, but also mathematical and economic notions of the term in general.
4
votes
1answer
199 views

Convergence of semi-gradient TD(0) with non-linear function approximation

I am looking for a result that shows the convergence of semi-gradient TD(0) algorithm with non-linear function approximation for on-policy prediction. Specifically, the update equation is given by (...
4
votes
1answer
96 views

REINFORCE algorithm for portfolio optimization - problem while training

I'm trying to implement the Reinforce algorithm (Monte Carlo policy gradient) in order to optimize a portfolio of 94 stocks on a daily basis (I have suitable historical data to achieve this). The idea ...
5
votes
1answer
50 views

How can we conclude that an optimization algorithm is better than another one

When we test a new optimization algorithm, what the process that we need to do?For example, do we need to run the algorithm several times, and pick a best performance,i.e., in terms of accuracy, f1 ...
0
votes
2answers
618 views

LSTM network doesn't converge, what should be changed? [closed]

I'm testing out TensorFlow LSTM layer text generation task, not classification task; but something is wrong with my code, it doesn't converge. What changes should be done? Source code: ...
1
vote
1answer
757 views

How is the actor-critic algorithm guaranteed to converge?

From my understanding, the critic evaluates the policy (actor) following dynamic programming (DP) or approximate dynamic programming (ADP) scheme, which should converge to the optimal value function ...
2
votes
0answers
43 views

How to show Monte Carlo methods converge to an estimate which minimizes mean squared error?

In chapter six of Sutton and Barto (p.128), they claim Monte Carlo methods converge to an estimate minimizing the mean squared error. How can this be shown formally? Bump