7

$\max(-y_i(w x_i), 0)$ is not partial derivable respect $w$ if $w x_i=0$. Loss functions are problematic when not derivable in some point, but even more when they are flat (constant) in some interval of the weights. Assume $y_i = 1$ and $w x_i < 0$ (that is, an error of type "false negative"). In this case, function $[y_i - \text{sign}(w x_i)]^2 ...


5

If the learning rate is greater than or equal to $1$ the Robbins-Monro condition $$\sum _{{t=0}}^{{\infty }}a_{t}^{2}<\infty\label{1}\tag{1},$$ where $a_t$ is the learning rate at iteration $t$, does not hold (given that a number bigger than $1$ squared becomes a bigger number), so stochastic gradient descent is not generally guaranteed to converge to a ...


3

Since we're dealing with real-values variables, it is almost certainly the case that the argument of the function will not be $0$. If you care strongly about that point, you can just use sub-gradients instead (and we do have sub-gradients for this function, so there is no problem).


3

The most usual case of bias=False is in layers before/after Batch Normalization with no activators in between. The BatchNorm layer will re-center the data anyway, removing the bias and making it a useless trainable parameter. Quoting the original BatchNorm paper: Note that, since we normalize $Wu+b$, the bias $b$ can be ignored since its effect will be ...


3

The first two equations are equivalent. The last equation can be equivalent if you scale $\alpha$ appropriately. Equation 1 Consider the equation from the Stanford slide: $$ v_{t}=\rho v_{t-1}+\nabla f(x_{t-1}) \\ x_{t}=x_{t-1}-\alpha v_{t}, $$ Let's evaluate the first few $v_t$ so that we can arrive at a closed form solution: $v_0 = 0 \\ v_1 = \rho v_0 + ...


3

Second-order optimization algorithms like Hessian optimization have more information on the curvature of the loss function, so converge much, much faster than first-order optimization algorithms like gradient descent. I remember reading somewhere that if you have $n$ weights in the neural network, one iteration of a second-order optimization algorithm will ...


2

I see why you might be confused. First, the logistic-loss or log-loss is technically called cross-entropy loss. This function is very simple: $CE = -[y \log(p) + (1 - y) \log(1 - p)]$ This tells basically if the predicted class $y$ was right $y=1$ then the loss is $CE=-\log(p)$, if the predicted class was not the right one then the loss is $CE=-\log(1-p)$. ...


2

You are correct. The main conceptual difference is that optimization is about finding the set of parameters/weights that maximizes/minimizes some objective function (which can also include a regularization term), while regularization is about limiting the values that your parameters can take during the optimization/learning/training, so optimization with ...


2

You might want to have a look at the wikipedia article of PCA, where it says: "The $k$th component can be found by subtracting the first $k − 1$ principal components from $\mathbf{X}$:" $$\hat{\mathbf{X}}_k = \mathbf{X} - \sum_{s=1}^{k-1}\mathbf{X}\mathbf{w}_s\mathbf{w}_s^T$$ Then you repeat the process to find the next component: $$\mathbf{w}_k = \...


1

You can define different measures in this way: Maximum distance of the new point with all points of the configuration ($M$) Minimum distance of the new point with all points of the configuration ($N$) $\frac{M}{\text{Maximum distance between two points of the configuration(}D)}$: normalize (1) measure $\frac{N}{D}$:normalize (2) measure You can get more ...


1

Note that you can't really predict whether your escape from a local minimum will work or not - you might just wind up in another, worse local minimum. The probability function you describe increases the likelihood of this happening. By upweighting the likelihood of allowing small energy differences, you allow for the possibility of escaping local minima, ...


1

Yes, you're interpreting the $\max$ there wrongly. In your second formula $$ \operatorname{Regret}_{T}(\mathcal{H})=\max _{h^{\star} \in \mathcal{H}} \operatorname{Regret}_{T}\left(h^{\star}\right) \label{1}\tag{1} $$ The sign $=$ means "is defined as", so maybe the following notation is less confusing $$ \operatorname{Regret}_{T}(\mathcal{H}) \...


1

The term REINFORCE actually corresponds to a method of estimating gradients, it is not particular to reinforcement learning. The paper you linked doesn't appear to deal with RL at all, so the issue they're describing is not one that you should expect to find in a policy gradient application. If you're using REINFORCE to estimate policy gradients in RL (this ...


1

That normal equation is sometimes called the closed-form solution. The short answer to your question is that the closed-form solution may be impractical or unavailable in certain cases or the iterative numerical method (such as gradient descent) may be more efficient (in terms of resources). This answer gives you more details and an example.


1

After diving deeper into the material I am able to answer my own question: Simulated Annealing tries to optimize a energy (cost) function by stochastically searching for minima at different temparatures via a Markov Chain Monte Carlo method. The stochasticity comes from the fact that we always accept a new state $c'$ with lower energy ($\Delta E < 0$), ...


1

The route/trajectory followed by the optimization algorithm basically depends your dataset and the loss function. However, what really matters, for the purpose of final accuracy performance, is the final point which the trajectory converges to.


1

In general it's better to not use sigmoid function in any hidden layer. There are many other great options such as ReLU and ELU. However, if for any reason you have to use sigmoid-like function, then go with Tanh function, at least it has ~0 mean.


1

There is a hardware based reasoning. Matrix multiplication is one of the central computations in deep learning. SIMD operations in CPUs happen in batch sizes, which are powers of 2. Here is a good reference about speeding up neural networks on CPUs by leveraging SIMD instructions: Improving the speed of neural networks on CPUs You will notice batch sizes ...


Only top voted, non community-wiki answers of a minimum length are eligible