Questions tagged [proofs]

For questions that ask about or call for proofs for specific assertions, whether they be proofs of theorems or corollaries, proofs of concept through working implementation, counter proofs, or counter examples.

Filter by
Sorted by
Tagged with
3 votes
1 answer
735 views

Why there are only three machine learning paradigms: supervised, unsupervised, reinforcement?

I read in books, blogs, and articles that there are three learning paradigms: supervised, unsupervised, and reinforcement. However, I have never found a proof that this list is exhaustive. Can it be ...
Vladislav Gladkikh's user avatar
0 votes
0 answers
14 views

Proof of Difference in Return Between Two Policies

I am attempting to understand why Lemma 6.1 holds in this paper on reinforcement learning. I have two questions. First, when defining the value function V(s), why is there a leading (1-γ) term? In the ...
Nikhil Sridhar's user avatar
1 vote
1 answer
31 views

Applicability of Holland's Schema Theorem to Genetic Algorithms with Non-Binary Individual Representations

I'm currently working on a problem formulation that requires non-binary individual representations in a genetic algorithm (GA). I've been exploring Holland's Schema Theorem as a theoretical basis for ...
CES's user avatar
  • 11
3 votes
1 answer
167 views

$\gamma^t$ in REINFORCE update (Sutton-Barto RL book Exercise 13.2)

I've struggled with solving exercise 13.2 from Reinforcement Learning: An Introduction Second Edition : Generalize the box on page 199, the policy gradient theorem (13.5), the proof of the policy ...
cfml's user avatar
  • 53
0 votes
1 answer
114 views

How to prove that $V^\star$ is optimal if and only if it satisfies the Bellman equation?

The Question I'd like to prove that a function $V$ (like in reinforcement learning) is optimal iff it satisfies the bellman equation. A lot of places online reference this fact, but none prove it. ...
snatchysquid's user avatar
1 vote
1 answer
85 views

Is there a mathematical proof that a binary neural network can approximate any function with arbitrary accuracy?

Since the Universal approximation theorem shows that standard multilayer feedforward networks with as few as a single hidden layer, sufficient hidden units, and arbitrary bounded and nonconstant ...
user68072's user avatar
1 vote
1 answer
795 views

Would it be possible to involve a proof assistant in the process of training a LLM?

LLMs like GPT-3 have been shown capable of outputting highly complex code. Sadly, actually using them to replace a programmer's job has two major caveats: LLMs are notoriously bad at producing ...
MaiaVictor's user avatar
0 votes
0 answers
27 views

How to prove that, for any graph $G$, the tree width of $G$ is $1$, if and only if $G$ is acyclic?

I am studying about CSPs, tree decomposition of a graph and tree width ($\text{TW}(G)$) of a graph - the smallest tree width of a tree decomposition. I encountered the following problem: Prove that ...
Daniel's user avatar
  • 131
5 votes
2 answers
154 views

Why is $\sum_{s} \eta(s)$ a constant of proportionality in the proof of the policy gradient theorem?

In Sutton and Barto's book (http://incompleteideas.net/book/bookdraft2017nov5.pdf), a proof of the policy gradient theorem is provided on pg. 269 for an episodic case and a start state policy ...
jwl17's user avatar
  • 59
1 vote
2 answers
126 views

Why $V^{\pi^*}(s) = \max_{a \in A}Q^{\pi^*}(s,a),\forall s \in S$ in reinforcement learning?

In some RL notes, I encountered the following equation, which I am trying to prove: $$ V^{\pi^*}(s) = \max_{a \in A}Q^{\pi^*}(s, a),\forall s \in S $$ Here is my attemption: Firstly, I only need to ...
vincen's user avatar
  • 11
2 votes
1 answer
766 views

Are my proofs that the Bellman operators are contractions correct?

Introduction I'm studying Reinforcement Learning, and in order to increase my understanding I've been challenging myself by trying to write proofs that show that the right hand side of the Bellman ...
cfml's user avatar
  • 53
1 vote
0 answers
26 views

Proof that the Policy Iteration Converges?

Let $\mathcal{X}=:\{x_1, x_2, x_3,...,x_n\}$ be the state space. Let $\mathcal{U}:=\{u_1, u_2, u_3,...,u_m\}$ be the set of actions. Let $A^{u_1}, A^{u_2}, A^{u_3},...,A^{u_m}$ be the state transition ...
SHASHANK RANJAN's user avatar
1 vote
1 answer
177 views

Are these two forms of the state value function the same?

Why are there different forms of the value function in reinforcement learning? Sutton & Barto (2nd edition, equation 3.14) define the state value function as follows $$v_{\pi}(s) = \displaystyle\...
cgo's user avatar
  • 175
3 votes
1 answer
181 views

Is there an error in A* optimality proof Russel-Norvig 4th edition?

In "AI: A Modern Approach", 4th edition, by Russell and Norvig, they give a purported proof that A* is cost-optimal for any admissible heuristic. The given proof seems most certainly wrong. ...
vdbuss's user avatar
  • 81
0 votes
0 answers
160 views

How to prove that "w will converge to TD fixed point once A is positive definite"

In Reinforcement Learning: An Introduction 2nd edition section 9.4 (p. 206), it says that when we use TD(0) as target and use semi-gradient method to update : In general, $w_t$ will be reduced ...
陳文澤's user avatar
1 vote
0 answers
210 views

Why does the schema theorem of genetic algorithms hold?

I have been reading about the Schema Theorem - one of the first theorems from the field of evolutionary computing and genetic algorithms, largely responsible for justifying the use of genetic ...
stats_noob's user avatar
1 vote
0 answers
165 views

Where do the characteristics of self-attention come into play in Linformer's proof that self-attention is low rank?

In Linformer's proof that self-attention is low rank in their paper, I don't see how it doesn't generalize to every matrix. They don't utilize any specifics of self-attention (the entire proof feels ...
mshlis's user avatar
  • 2,359
2 votes
0 answers
211 views

Why does TD (0) converge to the MLE solution of the Markov model?

Why does TD (0) converge to the MLE solution of the Markov model? Let's take the Example 6.4 in Sutton and Barto's book as an example. Example 6.4: You are the Predictor Place yourself now in the ...
XXX's user avatar
  • 143
0 votes
1 answer
50 views

How to show $\rho > 0$ when $\rho$ be minimum attainable from $y_n(W^{*T}X_n)$, where $W^*$ the vector that separates the data?

In the book Learning from Data written (by Abu Mostafa), we have the following exercise: Let $\rho$ be minimum attainable from $y_n(W^{*T}X_n)$ where $W^*$ is the vector that separates the data. Show ...
John Rawls's user avatar
2 votes
1 answer
242 views

What is the derivative of equation 1 in the paper "Conservative Q-Learning for Offline Reinforcement Learning"?

I am looking at the paper Conservative Q-Learning for Offline Reinforcement Learning, but I'm not sure how they proved theorem 3.1. Here is a screenshot of theorem 3.1. In the proof of theorem 3.1 ...
MoneyBall's user avatar
  • 121
0 votes
1 answer
93 views

Policies for which the policy improvement theorem holds

According to Reinforcement Learning (2nd Edition) by Sutton and Barto, the policy improvement theorem states that for any pair of deterministic policies $\pi'$ and $\pi$, if $q_\pi(s,\pi'(s)) \geq v_\...
bonzo_pippinpaddle's user avatar
2 votes
1 answer
54 views

How to prove importance sampling ratio is uncorrelated with action-value (or state-value) estimate?

In Sutton & Barto (2nd edition), the following is mentioned on page 150 (p. 172 of the pdf), section 7.4: the importance sampling ratio has expected value one (Section 5.9) and is uncorrelated ...
user529295's user avatar
0 votes
0 answers
31 views

Non-locally Electrically Programmable Logic Gates - Technological Advances Progress

Preface: I’d like to clarify that I understand what a relay is and that a PLC uses a fairly conventional microprocessor that only digitally establishes logical logic gate configuration as a digitally ...
Anony Mous's user avatar
3 votes
0 answers
97 views

How to prove Lemma 1.6 in the book "Reinforcement Learning: Theory and Algorithms"

I am trying to prove the following lemma from Reinforcement Learning: Theory and Algorithms on page 8. Lemma 1.6. We have that: $$ \left[(1-\gamma)\left(I-\gamma P^{\pi}\right)^{-1}\right]_{(s, a),\...
joesph chan's user avatar
6 votes
0 answers
98 views

Proof that there always exists a dominating policy in an MDP

I think that it is common knowledge that for any infinite horizon discounted MDP $(S, A, P, r, \gamma)$, there always exists a dominating policy $\pi$, i.e. a policy $\pi$ such that for all policies $\...
MMM's user avatar
  • 185
3 votes
1 answer
300 views

Why adding a baseline doesn't affect the policy gradient?

On the OpenAI's Spinning Up, they justify the fact that adding a baseline $b(s_t)$ in the policy gradient doesn't change its gradient by saying that this is an immediate consequence of the EGLP Lemma ...
Thomas Hustache's user avatar
4 votes
1 answer
141 views

How to prove the second form of Bellman's equation?

I'd like to prove this "second form" of Bellman's equation: $v(s) = \mathbb{E}[R_{t + 1} + \gamma v(S_{t+1}) \mid S_{t} = s]$ starting from Bellman's equation: $v(s) = \mathbb{E}[G_{t} \mid ...
Daviiid's user avatar
  • 573
2 votes
2 answers
358 views

Why is the equation $\mathbb{E} \left[ (Y - \hat{Y})^2 \right] = \left(f(X) - \hat{f}(X) \right)^2 + \operatorname{Var} (\epsilon)$ true?

In the book An Introduction to Statistical Learning, the authors claim (equation 2.3, p. 19, chapter 2) $$\mathbb{E} \left[ (Y - \hat{Y})^2 \right] = \left(f(X) - \hat{f}(X) \right)^2 + \operatorname{...
nbro's user avatar
  • 39.5k
1 vote
1 answer
81 views

When showing that the policy improvement theorem applies to MC control, why is $q_{\pi_{k}}\left(s, \pi_{k}(s)\right) \geq v_{\pi_{k}}(s)$ true?

When discussing why the policy improvement theorem is true, when we do Monte Carlo control by updating greedily, it says on page 98 of Sutton and Barto's book (2nd edition) that: $$ \begin{aligned} ...
Snowball's user avatar
  • 225
2 votes
1 answer
122 views

Is the existence and uniqueness of the state-value function for $\gamma < 1$ theoretical?

Consider the following statement from 4.1 Policy Evaluation of the first edition of Sutton and Barto's book. The existence and uniqueness of $V^{\pi}$ are guaranteed as long as either $\gamma < 1$...
hanugm's user avatar
  • 3,612
1 vote
0 answers
44 views

Why is the number of examined nodes $ O(b^{3d/4})$ in $\alpha$-$\beta$ pruning?

I'm taking a course 'Introduction to AI' and, in one of the tutorials, it was written that when pruning the game tree using $\alpha$-$\beta$ boundaries, the number of nodes that will be developed, ...
E. Ginzburg's user avatar
1 vote
0 answers
42 views

How to prove that a regularisation method simplified a neural network?

There are a few ways to regularise a neural network, for example dropout or the L1. Now, both these methods, and possibly most other regularisation methods, tend to remove from, or simplify the neural ...
Marcus's user avatar
  • 206
2 votes
1 answer
174 views

Is $\min(h_1(s),\ h_2(s))$ consistent?

If $h_1(s)$ is a consistent heuristic and $h_2(s)$ is a admissible heuristic, is $\min(h_1(s),\ h_2(s))$ consistent?
hello's user avatar
  • 21
0 votes
1 answer
923 views

Are hill climbing variations always optimal and complete?

Are hill climbing variations (like steepest ascent hill climbing, stochastic hill climbing, random restart hill climbing, local beam search) always optimal and complete?
user avatar
0 votes
0 answers
48 views

How is the discounted maximum entropy objective obtained for soft-q-learning and SAC

In the soft q-learning paper, they provide an expression for the maximum entropy objective that takes discounting into account. My main question is: can someone explain how they incorporated ...
quest ions's user avatar
1 vote
1 answer
100 views

Can the law of iterated expectation be used on the inner expectation of the DQN cost function described in the DQN paper

Is the expression for the DQN cost function, Equation (2) of the DQN paper $$\begin{align}L_1 &= E_{\mu,\pi}\left[\left(y_i - q(s,a;\theta)\right)^2\right]\\ &=E_{\mu,\pi}\left[\left(E_{\...
quest ions's user avatar
2 votes
1 answer
345 views

If $h_1(n)$ is admissible, why does A* tree search with $h_2(n) = 3h_1(n)$ return a path that is at most thrice as long as the optimal path?

Consider a heuristic function $h_2(n) = 3h_1(n)$. Where $h_1(n)$ is admissible. Why are the following statements true? $A^*$ tree search with $h_2(n)$ will return a path that is at most thrice as ...
Jia Rong's user avatar
7 votes
2 answers
2k views

How is this Pytorch expression equivalent to the KL divergence?

I found the following PyTorch code (from this link) -0.5 * torch.sum(1 + sigma - mu.pow(2) - sigma.exp()) where mu is the mean ...
user8714896's user avatar
3 votes
1 answer
407 views

Is my proof of equation 0.6 in the book "Reinforcement Learning: Theory and Algorithms" correct?

In Sham Kakade's Reinforcement Learning: Theory and Algorithms, this equation (page 17) is used preceding the proof of performance difference lemma. I am attempting to prove equation 0.6. Here is my ...
Isomorphism's user avatar
4 votes
0 answers
80 views

When do two identical neural networks have uncorrelated errors?

In Chapter 9, section 9.1.6, Raul Rojas describes how committees of networks can reduce the prediction error by training N identical neural networks and averaging the results. If $f_i$ are the ...
EmmanuelMess's user avatar
2 votes
1 answer
146 views

How to prove the formula of eligibility traces operator in reinforcement learning?

I don't understand how the formula in the red circle is derived. The screenshot is taken from this paper
hijkzzz's user avatar
  • 23
2 votes
0 answers
53 views

Does there necessarily exist "dominated actions" in a MDP?

In a Markov Decision Process, is it possible that there exists no "dominated action"? I define a dominated action the following way: we say that $(s,a)$ is a dominated action, if $\forall \...
Phil's user avatar
  • 31
1 vote
1 answer
258 views

How do I prove that the MSE is zero when all predictions are equal to the corresponding labels?

In the back-propogation algorithm, the error term is: $$ E=\frac{1}{2}\sum_k(\hat{y}_k - y_k)^2, $$ where $\hat{y}_k$ is a vector of outputs from the network, $y_k$ is the vector of correct labels (...
Slowat_Kela's user avatar
6 votes
2 answers
9k views

Is there a proof to explain why XOR cannot be linearly separable?

Can someone explain to me with a proof or example why you can't linearly separate XOR (and therefore need a neural network, the context I'm looking at it in)? I understand why it's not linearly ...
Slowat_Kela's user avatar
5 votes
2 answers
993 views

Given two optimal policies, is an affine combination of them also optimal?

If there are two different optimal policies $\pi_1, \pi_2$ in a reinforcement learning task, will the linear combination (or affine combination) of the two policies $\alpha \pi_1 + \beta \pi_2, \alpha ...
yang liu's user avatar
3 votes
2 answers
3k views

If uniform cost search is used for bidirectional search, is it guaranteed the solution is optimal?

If uniform cost search is used for both the forward and backward search in bidirectional search, is it guaranteed the solution is optimal?
user avatar
7 votes
1 answer
3k views

Why does a negative reward for every step really encourage the agent to reach the goal as quickly as possible?

If we shift the rewards by any constant (which is a type of reward shaping), the optimal state-action value function (and so optimal policy) does not change. The proof of this fact can be found here. ...
nbro's user avatar
  • 39.5k
3 votes
2 answers
218 views

Why does (not) the distribution of states depend on the policy parameters that induce it?

I came across the following proof of what's commonly referred to as the log-derivative trick in policy-gradient algorithms, and I have a question - While transitioning from the first line to the ...
stoic-santiago's user avatar
2 votes
0 answers
28 views

Do we assume the policy to be deterministic when proving the optimality?

In reinforcement learning, when we talk about the principle of optimality, do we assume the policy to be deterministic?
hakiki_makato's user avatar
1 vote
0 answers
98 views

How can I derive n-step off-policy temporal difference formula?

I was reading the book "Reinforcement Learning: An Introduction" by Sutton and Barto. In section 7.3, they write the formula for n-step off-policy TD as $$V(S_t) = V(S_{t-1}) + \alpha \rho_{...
Swakshar Deb's user avatar