Questions tagged [numerical-algorithms]

For questions about numerical algorithms used in artificial intelligence. Examples of those algorithms are Q-learning, simulated annealing, or any of the genetic algorithms. You should use this tag if you want to ask something about the numerical nature or analysis of those algorithms and you don't have more specific tags.

Filter by
Sorted by
Tagged with
0 votes
0 answers

For simple weight constraints: Add constraint directly or use parameterization without constraint

I am wondering if it makes sense to parameterize simple weight inequalities, for example if the weights should be $w\geq 0$, one cound train $\exp w$ over the unconstrained set instead. Also, if $\sum ...
0 votes
1 answer

How is catastrophic cancellation dealt with in loss functions?

It just occurred to me that this seems like it should be a very common problem that must have some kind of solution... Yet I'm not sure what it is... If there is no solution, does this mean once a ...
  • 296
1 vote
0 answers

GANs: Why does iterative gradient descent sometimes optimise $\min_G \max_D V(D,G)$ and sometimes $\max_D \min_G V(D,G)$?

For the following minimax equation for generative adversarial networks (GANs), $$\min_G \max_D V(D,G) = \mathbb{E}_{\boldsymbol{x}\sim p_{data}(\boldsymbol{x})}[\log D(\boldsymbol{x})] + \mathbb{E}_{\...
2 votes
3 answers

Is there any domain in machine learning that solves a problem by using only analytical algorithms?

Most of the algorithms in machine learning I am aware of use datasets and learning happens in an iterative manner given some examples. The examples can also be understood as experience in the case of ...
  • 3,251
3 votes
1 answer

What is numerical stability?

I came across the phrase "numerical stability" several times. But almost in the same context. I encountered this word mostly in the analytical formula for batch normalization. $$y = \dfrac{...
  • 3,251
2 votes
1 answer

Why is second-order backpropagation useful?

Raul Rojas's book on Neural Networks dedicates section 8.4.3 to explaining how to do second-order backpropagation, that is, computing the Hessian of the error function with respect to two weights at a ...
1 vote
1 answer

What are most commons methods to measure improvement rate in a meta-heuristic?

When I run a meta-heuristics, like a Genetic Algorithm or a Simulated Annealing, I want to have a termination criterion that stops the algorithms when there is not any significant fitness improvement. ...
  • 445
7 votes
1 answer

How is division by zero avoided when implementing back-propagation for a neural network with sigmoid at the output neuron?

I am building a neural network for which I am using the sigmoid function as the activation function for the single output neuron at the end. Since the sigmoid function is known to take any number and ...
  • 73