Questions tagged [convergence]

For questions related to the convergence of AI algorithms.

Filter by
Sorted by
Tagged with
1
vote
1answer
35 views

If deep Q-learning starts to choose only one action, is this a sign that the algorithm diverged?

I'm working on a deep q-learning model in an infinite horizon problem, with a continous state space and 3 possible actions. I'm using a neural network to approximate the action-value function. ...
3
votes
0answers
42 views

Convergence of a delayed policy update Q-learning

I thought about an algorithm that twists the standard Q-learning slightly, but I am not sure whether convergence to the optimal Q-value could be guaranteed. The algorithm starts with an initial ...
3
votes
1answer
67 views

What are the conditions of convergence of temporal-difference learning?

In reinforcement learning, temporal difference seem to update the value function in each new iteration of experience absorbed from the environment. What would be the conditions for temporal-...
2
votes
1answer
199 views

Why isn't my implementation of A2C for the the atari pong game converging?

I have two different implementations with PyTorch of the Atari Pong game using A2C algorithm. Both implementations are similar, but some portion are different. https://colab.research.google.com/drive/...
1
vote
0answers
19 views

Why is it hard to prove the convergence of the deep Q-learning algorithm?

Why is it hard to prove the convergence of the DQN algorithm? We know that the tabular Q-learning algorithm converges to the optimal Q-values, and with a linear approximator convergence is proved. ...
3
votes
1answer
53 views

Why do RL implementations converge on one action?

I have seen this happening in implementations of state-of-the-art RL algorithms where the model converges to a single action over time after multiple training iterations. Are there some general ...
1
vote
0answers
21 views

When does Monte Carlo linear function approximation converge?

In this Stanford lecture (minute 35:47 and 37:00), the professor says that Monte Carlo (MC) linear function approximation does not always converge, and she gives an example. In general, when does MC ...
0
votes
1answer
32 views

Is it a good idea to overfit on a small part of your data for faster model convergence?

I working on a classification problem that needs to detect patterns on a time serie. Basically, there's a catch-all class that means "no pattern detected", the other are for the specific patterns. The ...
2
votes
1answer
37 views

Neural network doesn't seem to converge with ReLU but it does with Sigmoid?

I'm not really sure if this is the sort of question to ask on here, since it is less of a general question about AI and more about the coding of it, however I thought it wouldn't fit on stack overflow....
1
vote
1answer
37 views

Why isn't the implementation of my policy evaluation for a simple MDP converging?

I am trying to code out a policy evaluation algorithm to find the $V^\pi(s)$ for all states. The following diagram below shows the MDP. In this case i let p = q = 0.5. the rewards for each states ...
2
votes
1answer
37 views

Is there an advantage in decaying $\epsilon$ during Q-Learning?

If the agent is following an $\epsilon$-greedy policy derived from Q, is there any advantage to decaying $\epsilon$ even though $\epsilon$ decay is not required for convergence ?
1
vote
0answers
21 views

Does SARSA(0) converge to the optimal policy in expectation if the Robbins-Monro conditions are removed?

The conditions of convergence of SARSA(0) to the optimal policy are : The Robbins-Monro conditions above hold for $α_t$. Every state-action pair is visited infinitely often The policy is greedy with ...
1
vote
2answers
36 views

What are the conditions for the convergence of SARSA to the optimal value function?

Is it correct that for SARSA to converge to the optimal value function (and policy) The learning rate parameter $\alpha$ must satisfy the conditions: $$\sum \alpha_{n^k(s,a)} =\infty \quad \text{and}...
0
votes
1answer
28 views

Is it possible to use deeplearning with spark (with a distributed databases as HDFS or Cassandra)?

If it is possible, will it be really useful or the model will end up converging very early(with a typical optimum learning rate) ? Any content on this topic will be helpful for me.
1
vote
1answer
61 views

Does TD(0) prediction require Robbins-Monro conditions to converge to the value function?

Does the learning rate parameter $\alpha$ require the Robbins-Monro conditions below for the TD(0) algorithm to converge to the true value function of a policy? $$\sum \alpha_t =\infty \quad \text{...
1
vote
1answer
150 views

Is there a simple proof of the convergence of TD(0)?

Does anybody know a simple proof of the convergence of the TD(0) value function prediction algorithm?
1
vote
0answers
25 views

Imposing contraints on sequence of image classifications

Are there example implementations of networks that apply constraints across sequences of image classifications where class labels are ordinal numbers? For example, to cause the output of a CNN to ...
4
votes
1answer
87 views

When exactly is a model considered over-parameterized?

When exactly is a model considered over-parameterized? There are some recent researches in Deep Learning about the role of over-parameterization toward generalization, so it would be nice if I can ...
2
votes
1answer
362 views

What is convergence in machine learning?

I came across this answer on Quora, but it was pretty sparse. I'm looking for specific meanings in the context of machine learning, but also mathematical and economic notions of the term in general.
4
votes
1answer
81 views

Convergence of semi-gradient TD(0) with non-linear function approximation

I am looking for a result that shows the convergence of semi-gradient TD(0) algorithm with non-linear function approximation for on-policy prediction. Specifically, the update equation is given by (...
3
votes
1answer
57 views

REINFORCE algorithm for portfolio optimization - problem while training

I'm trying to implement the Reinforce algorithm (Monte Carlo policy gradient) in order to optimize a portfolio of 94 stocks on a daily basis (I have suitable historical data to achieve this). The idea ...
5
votes
1answer
42 views

How can we conclude that an optimization algorithm is better than another one

When we test a new optimization algorithm, what the process that we need to do?For example, do we need to run the algorithm several times, and pick a best performance,i.e., in terms of accuracy, f1 ...
0
votes
2answers
246 views

LSTM network doesn't converge, what should be changed? [closed]

I'm testing out TensorFlow LSTM layer text generation task, not classification task; but something is wrong with my code, it doesn't converge. What changes should be done? Source code: ...
1
vote
1answer
266 views

How is the actor-critic algorithm guaranteed to converge?

From my understanding, the critic evaluates the policy (actor) following dynamic programming (DP) or approximate dynamic programming (ADP) scheme, which should converge to the optimal value function ...
2
votes
0answers
42 views

How to show Monte Carlo methods converge to an estimate which minimizes mean squared error?

In chapter six of Sutton and Barto (p.128), they claim Monte Carlo methods converge to an estimate minimizing the mean squared error. How can this be shown formally? Bump
3
votes
1answer
186 views

How to show temporal difference methods converge to MLE?

In chapter 6 of Sutton and Barto (p. 128), they claim temporal difference converges to the maximum likelihood estimate (MLE). How can this be shown formally?
1
vote
1answer
76 views

Is there a rigorous proof for finding Hopfield minima?

I am looking for a rigorous mathematical proof for finding the several local minima of the Hopfield networks. I am searching for something rigorous, a demonstration, not just let the network keep ...
3
votes
0answers
29 views

How do we give a kick start to the Facenet network?

I read the Facenet paper and one thing I am not sure about (it might be trivial and I missed it) is how do we give the kick start to the network. The embeddings in the beginning are random, so ...
3
votes
0answers
36 views

How to know when a Environment will yield a deterministic model

Given enough experiment data on time taken for objects to fall to earth from different heights, one can create various models that will accurately predict the time it will take for an object falling ...
2
votes
0answers
26 views

Are there reinforcement learning algorithms that ensure convergence for continuous state space problems?

The Q-learning does not guarantee convergence for continuous state space problems (Why doesn't Q-learning converge when using function approximation?). In that case, is there an algorithm which ...
1
vote
1answer
57 views

DQN Q-values are static

I am working on a DDQN with 5 LSTM layers and 3 actions as output and state space of 21 features. I am dividing the dataset into episodes of 720 timesteps, for each episode the agent acts greedily for ...
1
vote
1answer
231 views

LSTM is not converging

I am writing my first LSTM network and I would really appreciate if someone can tell me if it is right (the loss seems to go down very slowly and before playing around with hyper parameters I want to ...
-2
votes
1answer
1k views

RNN LSTM not converging with Adam

I am trying to train a RNN with text from wikipedia but I having having trouble getting the RNN to converge. I have tried increasing the batch size but it doesn't seem to be helping. All data is one ...
2
votes
1answer
495 views

Deep Q-Learning poor convergence on Stochastic Environment

I'm trying to implement a Deep Q-network in Keras/TF that learns to play Minesweeper (our stochastic environment). I have noticed that the agent learns to play the game pretty well with both small and ...
2
votes
1answer
94 views

Is a calculus or ML approach to varying learning rate as a function of loss and epoch been investigated?

Many have examined the idea of modifying learning rate at discrete times during the training of an artificial network using conventional back propagation. The goals of such work have been a balance ...
2
votes
3answers
184 views

What is chaotic behavior and how it is achieved in non-linear regression and artificial networks?

I'm finding it hard to understand the relationship between chaotic behavior, the human brain, and artificial networks. There are a number of explanations on the web, but it would be very helpful if I ...
0
votes
1answer
63 views

Which is more important, doubt or reinforcement?

Reinforcement? We hear much about reinforcement, which is, in my opinion a poor choice of a term to describe a type of artificial network that continues to acquire or improve its behavioral ...