8

To understand this equation first you need to understand the context in which it is first introduced. We have two neural networks (i.e. $D$ and $G$) that are playing a minimax game. This means that they have competing goals. Let's look at each one separately: Generator Before we start, you should note that throughout the whole paper the notion of the data-...


7

In general the different reward functions $R(s)$, $R(s, a)$ and $R(s, a, s')$ are not equivalent mathematically, so you will not find any formal proof. It is possible for the functions to resolve to the same value in a specific MDP, if, for instance, you use $R(s, a, s')$ and the value returned only depends on $s$, then $R(s, a, s') = R(s)$. This is not true ...


6

This expression: $|\mathcal{A}(s)|$ means $|\quad|$ the size of $\mathcal{A}(s)$ the set of actions in state $s$ or more simply the number of actions allowed in the state. This makes sense in the given formula because $\frac{\epsilon}{|\mathcal{A}(s)|}$ is then the probability of taking each exploratory action in an $\epsilon$-greedy policy. The overall ...


5

$\|x\| = |x|$ denotes the absolute value norm, which is a special case of the $L_1$ norm defined on the 1-D vector spaces formed by real or complex numbers. $\|\textbf{x}\|_1 = \sum_{i=1}^n|x_i|$ denotes the Taxicab / Manhattan norm, relating to how a Taxi would drive along a rectangular grid of roads to reach a point $(x, y)$ from $(0,0)$. $\|\textbf{x}\|...


5

The first part of this answer is a little background that might bolster your intuition for what's going on. The second part is the more practical and direct answer to your question. The gradient is just the generalization of the derivative to multivariable functions. The gradient of a function at a certain point is a vector that points in the direction of ...


5

When lambda = 0 as in TD(0), how does the method learn? As it appears, with lambda = 0, there will never be a change in weight and hence no learning. I think the detail that you're missing is that one of the terms in the sum (the final "iteration" of the sum, the case where $k = t$) has $\lambda$ raised to the power $0$, and anything raised to the power $0$...


5

It means that $z$ has a (multivariate) normal distribution with 0 mean and identity covariance matrix. This essentially means each individual element of the vector $z$ has a standard normal distribution.


4

Here is a paper with the mathematical definition of each term: Let Nt,n,σ,L be all target functions that can be implemented using a neural network of depth t, size n, activation function σ, and when we restrict the input weights of each neuron to be |w|1 + |b| ≤ L.


4

Let $R(s)$ denote a probability distribution over rewards that our agent may get in some MDP as a reward for entering a state $s$. The easiest case is to demonstrate that we can also choose to write this as $R(s, a)$ or $R(s, a, s')$: simply take $\forall a: R(s, a) = R(s)$, or $\forall a \forall s': R(s, a, s') = R(s)$, as also described in Neil's answer. ...


4

In the Sutton and Barto book $q(s,a)$ is used to denote the true expected value of taking action $a$ in state $s$, whereas capital $Q(s,a)$ is used to denote an estimate of $q(s,a)$. However, there is likely to be a lot of inconsistency in the literature as each author has their own preference on how to denote things. I would encourage you to consider ...


3

It doesn't seem that it is a "proper" symbol. I guess that $\sup$ simply refers to the supremum, that is, you want to select actions that maximize the quantity that comes to the right of $\sup$, while $\text{dist}$ is simply a proxy for any possible distance between distributions. For example, you can replace $\text{dist}$ with the Kullback-Leibler ...


3

Random variables You do not necessarily need to understand the concept of a random variable (r.v.) to understand the concept of a probability distribution, but the concept of a random variable is strictly connected to the concept of a probability distribution (given that each random variable has an associated probability distribution), so, before proceeding, ...


3

This is standard backpropagation. The gradient term you see is in fact a vector of partial derivatives where each element is the partial derivative of the log-likelihood with respect to each element of the parameter vector $\theta$. Therefore, it has the same dimensionality as $\theta$. Each element of the parameter vector is then updated with the respective ...


3

No, the substitution you suggest based on Equation (3.4) is not correct because you forgot about the $\sum_{r \in \mathcal{R}}$ in the right-hand side Equation (3.4). Equation (3.4) says (leaving out the middle part): $$p(s' \vert s, a) \doteq \sum_{r \in \mathcal{R}} p(s', r \vert s, a).$$ If you plug this into Equation (3.6) to substitute the ...


3

At page 130 of the same book, the author states that $\hat{p}_\text{data}$ is an empirical distribution defined by the training data. Similarly, at page 129, he states that $p_\text{data}$ is the true distribution that generates the set $\mathbb{X} = \{ \boldsymbol{x}^{(1)}, \dots, \boldsymbol{x}^{(m)} \}$. What is the difference between $\hat{p}_\text{data}...


3

The square brackets $[]$ in $[\tau_{ij}]^\alpha$ and $[\eta_{ij}]^\beta$ may be just a way of emphasing that the elements $\tau_{ij} \in \mathbb{R}$ and $\eta_{ij} \in \mathbb{R}$ of respectively the matrices $\mathbf{\tau} \in \mathbb{R}^{n \times n}$ and $\mathbf{\eta} \in \mathbb{R}^{n \times n}$ (where $n$ is the number of nodes in the graph) are ...


3

I am using the convention of uppercase $X$ for random variable and lowercase $x$ for an individual observation. It is possible your source material did not do this, which might be causing your confusion. However, it is the convention used in Sutton & Barto's Reinforcement Learning: An Introduction. What I didn't understand what is 𝑋 here. i.e., what is ...


3

Whilst you're right that for any continuous distribution $P(X = x) = 0 \;; \forall x \in \mathcal{X}$ where $\mathcal{X}$ is there support of the distribution, they are not referring to probabilities here, rather they are referring to density functions (though this should really be denoted with a lower case $p$ to avoid confusion such as this). $p(x|z)$ is a ...


2

So, what is the purpose of the new index for $V$ in Chapter 7, and why is it more important at this particular chapter? My guess would be that your intuition is correct, and that it's mostly introduced just to clarify exactly which "version" of our value function approximator is going to be used in any particular equation. In previous chapters, which ...


2

This is a commonly used notation in theoretical computer science. $[m]$ is not the variable $m$, but is instead the set of integers from $1$ to $m$ inclusive. The empirical error equation thus reads in English: The cardinality of a set consisting of the elements $i$ of the set of integers $[m]$ such that the hypothesis given input $x_i$ disagrees with ...


2

$$1-\sum_i(e_i-a_i)^2$$ $\sum$ - there just means sum. It is the greek letter for S. You can rewrite the above formula as $$1 -[(e_1 - a_1)^2+(e_2-a_2)^2+(e_3-a_3)^2+\ldots ]$$ $\sum$ just helps us avoid writing dozens of $+$ signs. Read more here. What they are doing here is taking the difference of expected value $e_1$ and the actual value $a_1$ for the ...


2

Your first option is correct: $$r(s,a) = \mathbb{E}\left[R_t|S_{t-1}=s,A_{t-1}=a\right]=\sum_{r\in \mathcal{R}}\left[r\sum_{s'\in \mathcal{S}}p(s',r|s,a)\right]$$ It's partly a matter of taste, but I prefer not moving the $r$ into the double sum, because its value does not change in the "inner loop". There is a small amount of intuition to be had that way ...


2

This equation and more information of it can be found in Expectation Maximization Wikipedia site and the explanation there was as follows (formula there in two parts): Some more explanation from same page: In statistics, an expectation–maximization (EM) algorithm is an iterative method to find maximum likelihood or maximum a posteriori (MAP) estimates of ...


2

A probability distribution in ML is the same as a probability distribution elsewhere. A probability distribution (or probability function, or probability mass function, or probability density function) is any function that accepts as input elements of some specific set $x \in X$, and produces as output, real-valued numbers between 0 and 1 (inclusive), such ...


2

In full: The limit, as standard deviation $\sigma$ tends towards zero, of the gradient with respect to vector $\mathbf{x}$, of the expectation - where perturbation $\epsilon$ follows the normal distribution with mean 0 and variance $\sigma^2$ times identity vector $[1,1,1,1...]$ * - of any function $f$ of $\mathbf{x}$ plus $\epsilon$ is equal to the ...


2

The dot ($.$) at the end of $T(s,a,.)$ shows all possible states that we can go from state $S$ by doing action $a$. As you know there are some probabilities here for choosing those states, that the sum of these probabilities is equal to 1. Hence, $T(s,a,.)$ is a probability distribution.


2

I assume this is an iterative function. It means the current $V(S_t)$ is the sum of the previous plus some adjustment. The arrow is like an assignment. In code, you would do vst = vst + alpha * (gt - vst) So vst will be overwritten.


2

In the source code, the author defines sd by sd = 0.5 * tf.layers.dense(x, units=n_latent) which means that $\operatorname{sd}\in \mathbb{R}^n$. In particular, the support over sd includes negative numbers, which is something we want to avoid. Since standard deviations are always nonnegative, we can exponentiate to get us in the correct domain. ...


2

The $\sim$ symbol means that a random variable is drawn from the given distribution, i.e. if I were to say $X$ has a Standard Normal distribution I would write $X \sim \text{Normal}(0,1)$. They write two explicit expectations here because $a$ is a random variable with distribution $\mu_x$ but $X$ is also a random variable with distribution $V$. I believe you ...


2

Isn't the distribution independent of the time the arm $i$ was chosen? Yes, but you don't know which arm was chosen at time $t$, that is what $I_t$ represents. $v_i$ would represent the $i$th arms distribution, whereas you want the distribution of the arm that was chosen at time $t$, which is $v_{I_t}$. $X_{I_t,t}$ is used to represent the arm you chose at ...


Only top voted, non community-wiki answers of a minimum length are eligible