Questions tagged [optimization]

For questions about implementing and improving optimization algorithms used in creating AI programs, or optimization in general.

Filter by
Sorted by
Tagged with
12
votes
1answer
4k views

Loss jumps abruptly when I decay the learning rate with Adam optimizer in PyTorch

I'm training an auto-encoder network with Adam optimizer (with amsgrad=True) and ...
11
votes
1answer
665 views

What are Hyper-heuristics?

I wanted to know what the differences between hyper-heuristics and meta-heuristics are, and what their main applications are. Which problems are suited to be solved by Hyper-heuristics?
10
votes
2answers
288 views

Can artificial intelligence be thought of as optimization?

In this video an expert says, "One way of thinking about what intelligence is [specifically with regard to artificial intelligence], is as an optimization process." Can intelligence always be thought ...
10
votes
1answer
700 views

What are the implications of the “No Free Lunch” theorem for machine learning?

The No Free Lunch (NFL) theorem states (see the paper Coevolutionary Free Lunches by David H. Wolpert and William G. Macready) any two algorithms are equivalent when their performance is averaged ...
9
votes
2answers
10k views

What are the limitations of the hill climbing algorithm and how to overcome them?

What are the limitations of the hill climbing algorithm? How can we overcome these limitations?
8
votes
2answers
2k views

Why number of hidden units in a layer are suggested to be in powers of 2?

It is suggested that the number of hidden units in a layer should be in powers of 2 because it helps converge faster. Is it a fact and if it is, how this helps the NN learn faster. Does it have to do ...
8
votes
2answers
285 views

Why does a one-layer hidden network get more robust to poor initialization with growing number of hidden neurons?

In a nutshell: I want to understand why a one hidden layer neural network converges to a good minimum more reliably when a larger number of hidden neurons is used. Below a more detailed explanation of ...
8
votes
1answer
637 views

Why does 'loss' change depending on the number of epochs chosen?

I am using Keras to train different NN. I would like to know why if I increment the epochs in 1, the result until the new epoch is not the same. I am using shuffle=False, and np.random.seed(2017), and ...
7
votes
3answers
126 views

How much can the addition of new features improve the performance?

How much can the addition of new features improve the performance of the model during the optimization process? Let's say I have a total of 10 features. Suppose I start the optimisation process using ...
6
votes
2answers
68 views

Why perceptron criterion function is differentiable?

I'm reading the chapter one of the book called "Neural Networks and Deep Learning" from Charu C. Aggargal. In the section 1.2.1.1 of the book, I'm learning about the perceptron. One thing ...
6
votes
1answer
648 views

When should we use algorithms like Adam as opposed to SGD?

As far as I know, Stochastic Gradient Descent is an optimization algorithm which belongs to the the category of algorithms where hyper-parameters have to be defined beforehand. They are useful in many ...
5
votes
3answers
843 views

What is the actual learning algorithm: back-propagation or gradient descent?

What is the actual learning algorithm: back-propagation or gradient descent (or, in general, the optimization algorithm)? I am reading through chapter 8 of Parallel Distributed Processing hand book ...
5
votes
1answer
3k views

How to avoid falling into the “local minima” trap?

How do I avoid my gradient descent algorithm into falling into the "local minima" trap while backpropogating on my neural network? Are there any methods which help me avoid it?
5
votes
1answer
79 views

What are the techniques for detecting and preventing overfitting?

I'm worrying that my neural network has become too complex. I don't want to end up with half of the neural network doing nothing but just take up space and resources. So, what are the techniques for ...
5
votes
1answer
2k views

How can we use linear programming to solve an MDP?

Apparently, we can solve an MDP (that is, we can find the optimal policy for a given MDP) using a linear programming formulation. What's the basic idea behind this approach? I think you should start ...
5
votes
1answer
91 views

What are examples of optimization problems that can be solved using genetic algorithms?

I'm trying to learn how genetic algorithms can solve optimization problems. I have already learned how genetic algorithms can solve the knapsack, TSP and set cover problems. I'm looking for some other ...
5
votes
1answer
46 views

How can we conclude that an optimization algorithm is better than another one

When we test a new optimization algorithm, what the process that we need to do?For example, do we need to run the algorithm several times, and pick a best performance,i.e., in terms of accuracy, f1 ...
5
votes
1answer
120 views

Reproduce Firefly Algorithm experiments of original paper?

I have been trying to reproduce the experiments done in the original: "Firefly Algorithm for multimodal optimization" (linked in the question) so far: unsuccessfully. For the moment being I'm okay if ...
4
votes
3answers
120 views

Can we optimize an optimization algorithm?

In this answer to the question Is an optimization algorithm equivalent to a neural network?, the author stated that, in theory, there is some recurrent neural network that implements a given ...
4
votes
1answer
920 views

What's the difference between RMSE and Euclidean distance, and when to use a custom loss? [closed]

I'm searching for a loss function that fits my project. Actually, I have two questions, but they are in the same direction. I take a look at the definition of the root mean squared error (RMSE) and ...
4
votes
1answer
69 views

Can we use the Tierra approach to optimize machine code?

Thomas Ray's Tierra is a computer program which simulates life. In the linked paper, he argues how this simulation may have real-world applications, showing how his digital organisms (computer ...
4
votes
2answers
847 views

When should I use simulated annealing as opposed to a genetic algorithm?

What kind of problems is simulated annealing better suited for compared to genetic algorithms? From my experience, genetic algorithms seem to perform better than simulated annealing for most problems....
4
votes
1answer
117 views

What effect does batch norm have on the gradient?

Batch norm is a technique where they essentially standardize the activations at each layer, before passing it on to the next layer. Naturally, this will affect the gradient through the network. I have ...
4
votes
1answer
109 views

What is the basic purpose of local search methods?

I read about the hill climbing algorithms, the simulating annealing algorithm, but I am confused. What is the basic purpose of local search methods?
4
votes
1answer
41 views

How to use MOPSO to align characters vertically?

I need to efficiently align characters vertically using Multi Objective PSO. Alignment is achieved by adding spaces in between a given set of characters. ...
3
votes
2answers
71 views

Is there a way to define the boundaries of the optimal size of a training set?

At a related question in Computer Science SE, a user told: Neural networks typically require a large training set. Is there a way to define the boundaries of the "optimal" size of a training set ...
3
votes
2answers
67 views

Could error surface shape be useful to detect which local minima is better for generalization?

The following plot shows error function output based on system weights. Two equal local minima are shown in green pointers. Note that the red dots are not related to the question. Does the right one ...
3
votes
1answer
54 views

Maximizing or Minimizing in Trust Region Policy Optimization?

I happened to discover that the v1 (19 Feb 2015) and the v5 (20 Apr 2017) versions of TRPO papers have two different conclusions. The Equation (15) in v1 is $\min_\theta$ while the Equation (14) in v2 ...
3
votes
1answer
66 views

Which algorithm can I use to solve a problem with multiple objectives and constraints?

Consider a problem with many objectives. In my case, these are school grades for different courses (or subjects). To be more concrete, suppose that my current grade for the math course is $12/20$ and ...
3
votes
1answer
122 views

Is it possible to have a dynamic $Q$-function?

I am trying to use Q-learning for energy optimization. I only wish to have states that will be visited by the learning agent, and, for each state, I have a function that generates possible actions, so ...
3
votes
1answer
247 views

Why does the hill climbing algorithm only produce a local maximum?

Apparently, the hill climbing algorithm just produces a local maximum, and not necessarily a global optimum. It's stuck on a local maximum. Why does hill climbing algorithm only produce a local ...
3
votes
1answer
87 views

Genetic Algorithms: Trade-off between time and variance with regards to fitness function

I'm developing an AI to play a card game with a genetic algorithm. Initially, I will evaluate it against a player that plays randomly, so there will naturally be a lot of variance in the results. I ...
3
votes
2answers
602 views

Input optimization on a supervised learning system

Problem Given a collection of pairs (X, y) where X belongs to R^n and y belongs to R, find the X such that the associated y would be maximum. Example Given: (X=(1, 2), y=-9) (X=(-2, 4), y=-36) (X=...
3
votes
1answer
260 views

Problems getting ADADELTA to converge

I have followed the pseudocode in the ADADELTA paper (top right on page 3), and wrote the following Python code for solving the optimization problem L(x) = x^2: ...
3
votes
1answer
41 views

Measuring novel configuration of points

I am trying to implement Novelty search; I understand why it can work better than the standard Genetic Algorithm based solution which just rewards according to the objective. I am working on a problem ...
3
votes
1answer
746 views

Advantages of Kullback-Leibler over L1/L2?

I've recently encounter different articles that are recommending to use KL instead of L1/L2 norm when trying to minimize a probability distribution. But none of the articles are giving a clear ...
3
votes
0answers
77 views

How does SGD escape local minima?

SGD is able to jump out of local minima that would otherwise trap BGD I don't really understand the above statement. Could someone please provide a mathematical explanation for why SGD (Stochastic ...
3
votes
2answers
440 views

In deep learning, is it possible to use discontinuous activation functions?

In deep learning, is it possible to use discontinuous activation functions (e.g. one with jump discontinuity)? (My guess : for example, ReLU is non-differentiable at a single point, but it still has ...
3
votes
0answers
75 views

Does Retina-net's focal loss accomplish its goal?

Taking out the weighting factor we can define focal loss as $$FL(p) = -(1-p)^\gamma log(p) $$ Where $p$ is the target probability. The idea being that single stage object detectors have a huge ...
3
votes
0answers
88 views

Which features and algorithm could optimize this air-conditioner problem?

Imagine we have 2 air conditioner systems (AA) and 2 "free cooling" systems which mix external and internal air (FC) in a closed box which always tends to warm up. For each system, we have to find ...
3
votes
0answers
37 views

Knapsack of mixture with constraints

I'm trying to find the optimized mixture for a specific set of substances. Each of those substances have characteristics that I want to optimize in the mixture (some characteristics I want to minimize ...
2
votes
1answer
148 views

If Deep Learning is non convex, then why use convex loss?

I was just reading through some convex optimization textbooks to hopefully improve my deep learning understanding and come up with new ideas. Halfway through, I decided to Google a bit! It's obvious ...
2
votes
2answers
52 views

How can I train a neural network if I don't have enough data?

I have created a neural network that is able to recognize images with the numbers 1-5. The issue is that I have a database of 16x5 images which ,unfortunately, is not proving enough as the neural ...
2
votes
1answer
65 views

Method to check goodness of combinatorial optimization algorithm implementation

How do I check which algorithm solves my problem best? Given a optimaization problem, I apply different well known optimization algorithms (genetic algorithm, simulated annealing, ant colony etc.) to ...
2
votes
3answers
90 views

Would a sentient AI try to create a more optimised AI which would eventually overtake AI 1.0?

Would AI be a self-propogating iteration in which the previous AI is destroyed by a more optimised AI child? Would the AI have branches of it's own AI warning not to create the new AI?
2
votes
1answer
67 views

How are these equations of SGD with momentum equivalent?

I know this question may be so silly, but I can not prove it. In Stanford slide (page 17), they define the formula of SGD with momentum like this: $$ v_{t}=\rho v_{t-1}+\nabla f(x_{t-1}) \\ x_{t}=x_{...
2
votes
1answer
93 views

Why is second-order backpropagation useful?

Raul Rojas's book on Neural Networks dedicates section 8.4.3 to explaining how to do second-order backpropagation, that is, computing the Hessian of the error function with respect to two weights at a ...
2
votes
1answer
99 views

Is a calculus or ML approach to varying learning rate as a function of loss and epoch been investigated?

Many have examined the idea of modifying learning rate at discrete times during the training of an artificial network using conventional back propagation. The goals of such work have been a balance ...
2
votes
2answers
109 views

How does PCA work when we reduce the original space to 2 or higher-dimensional space?

How does PCA work when we reduce the original space to a 2 or higher-dimensional space? I understand the case when we reduce the dimensionality to $1$, but not this case. $$\begin{array}{ll} \text{...
2
votes
2answers
102 views

Solving a planning if finding the goal state is part of the problem

I having trouble finding some starting points for solving an occupancy problem which seems like a good candidate for ai. Assume the following situation: In a company I have n cars and m employees. ...