Questions tagged [optimization]

For questions about implementing and improving optimization algorithms used in creating AI programs, or optimization in general.

Filter by
Sorted by
Tagged with
0
votes
0answers
21 views

What I Should Do to Reduce Solution Size for Simulated Annealing Algorithm?

I am trying to find the best solution for radar placement problem with using multi objective simulated annealing algorithm. So there is an area (in real map) and I want to put minimum count of radar ...
1
vote
0answers
72 views

Why does the schema theorem of genetic algorithms hold?

I have been reading about the Schema Theorem - one of the first theorems from the field of evolutionary computing and genetic algorithms, largely responsible for justifying the use of genetic ...
1
vote
1answer
40 views

Does reaching the global optima guarantee good performance in a task?

It is to my understanding that, in deep learning, we are essentially trying to minimize the loss function that we have defined and reach its global optima through some form of optimization technique. ...
0
votes
0answers
30 views

Limit of momentum update equation

I am self-studying on optimization algorithm on https://d2l.ai/chapter_optimization/momentum.html and couldn't get my head around some derivation: Instead of the standard gradient descent update ...
0
votes
0answers
45 views

Is it possible to find a good neural network structure without training it? [duplicate]

Neural networks consist of so many parameters. Researchers could create as many possible neural networks as they wish. So I want to ask a general question. Could we devise an evolutionary algorithm ...
1
vote
0answers
20 views

a loss for binary step function data

I have some data with ground truth that looks like a binary step function, where part of it is 0 and part is one. An example for the GT can be like ...
0
votes
0answers
24 views

Why is there a Hessian diagonal approximation? And when can we use it?

This topic has been introduced in "Pattern Recognition and Machine Learning, Bishop, 2006", section 5.4.1. I am a bit confused about this method and I have two questions. Why this method ...
0
votes
0answers
20 views

How to smooth a cost function?

I have a combinatorial optimization problem whose loss function is really unsmooth now. Without specifying my problem in detail, I was wondering if there are some general methods/steps I can apply to ...
2
votes
1answer
74 views

How do I use machine learning to create an optimization algorithm?

Let's say that I want to create an optimization algorithm, which is supposed to find an optimum value for a given objective function. Creating an optimization algorithm to explore through the search ...
0
votes
0answers
18 views

Is there a way to adapt Particle Swarm Optimization to an incremental/online learning setting?

As stated in the title, is there a way to adapt PSO to an online scenario where new data samples arrive continuously? In more detail: suppose that I have a classifier with several parameters for which ...
2
votes
1answer
70 views

Are Genetic Algorithms suitable for a problem with a non-unique optimal solution?

I was wondering if a genetic algorithm is useful if the optimization problem has several optimal solutions. My thought was that I should not use it since when combining two members of a population who ...
4
votes
2answers
215 views

Why is gradient descent used over the conjugate gradient method?

Based on some preliminary research, the conjugate gradient method is almost exactly the same as gradient descent, except the search direction must be orthogonal to the previous step. From what I've ...
1
vote
0answers
34 views

Multi-armed Bandit in optimization on graph edges selection

I have the problem, which I described below. I wonder if there exists a class of multi-armed bandit approaches that is related to it. I am working on computer networking optimization. In the simplest ...
1
vote
0answers
43 views

Why are optimization algorithms for deep learning so simple?

From my knowledge, the most used optimizer in practice is Adam, which in essence is just mini-batch gradient descent with momentum to combat getting stuck in saddle points and with some damping to ...
0
votes
1answer
159 views

What is uncentered variance and how it becomes equal to mean square in Adam?

I have been reading about Adam and AdamW (Here). The author mentioned that in "uncentered variance" we don't consider subtracting mean In this statement, the author is talking about ...
1
vote
0answers
26 views

How do you decide that you have tested enough hyper-parameter combinations for a specific neural network architecture?

How do you decide that you have tested enough hyper-parameter combinations for a specific neural network architecture to discard it and move on to a new model? Do you have a structured (generic) ...
0
votes
1answer
32 views

Compare the efficiency of a trained ML model with a non-learning-based method for solving the same problem

If a certain task T is solved by a non-learning-based method A (let's say, an optimization-based approach). We now train a machine learning model B (let's say a neural network) on the same task. What ...
0
votes
0answers
29 views

How do I solve a minimization problem with Q-learning learning?

I am trying to learn reinforcement learning by myself and so I have a lot of doubts. In particular, I am investigating how to use Q-learning in order to solve minimization problems. For example, ...
1
vote
1answer
31 views

Which class of functions are quite complicated in deep learning?

Deep learning is a field in which we need neural networks that are deep enough to carry on our task. The important fucntions in deep neural networks can be classified in to three classes: activation ...
1
vote
1answer
28 views

Do gradient-based algorithms deal with the flat regions with desired points?

I am studying a chapter named Numerical Computation of a deep learning book. Afaik, it does not deal with flat regions with desired points. For example, let us consider a function whose local/global ...
0
votes
1answer
66 views

How many directions of gradients exist for a function in higher dimensional space?

Gradients are used in optimization algorithms. Based on the values of gradients, we generally update the weights of a neural network. It is known that gradients have a direction and the direction ...
0
votes
0answers
33 views

What all does the gradient tells us?

Gradients are used in optimization algorithms. I know that a gradient gives us information about the direction in which one needs to update the weights of a neural network. We need to travel in the ...
0
votes
2answers
33 views

Reason for relaxing limit in derivative in this context?

Consider the following paragraph from NUMERICAL COMPUTATION of the deep learning book.. Suppose we have a function $y = f(x)$, where both $x$ and $y$ are real numbers. The derivative of this function ...
0
votes
0answers
12 views

What is meant by non-convergent limit cycles?

Limit cycle is a closed curve that is isolated i.e., no other closed curve near to it. You can read the following paragraph from here If there is (such) a closed curve, the nearby trajectories must ...
1
vote
1answer
27 views

Cover a surface with smaller predefined objects

I'm trying to make a program that takes a surface designed by the user, and different 3D geometries from a dataset as inputs and gives a good approximation of the surface using only the objects found ...
3
votes
1answer
64 views

What are the necessary mathematical properties to be a loss function in gradient based optimizations?

Loss functions are used in training neural networks. I am interested in knowing the mathematical properties that are necessary for a loss function to participate in gradient descent optimization. I ...
1
vote
0answers
15 views

What does the lambda parameter in the paper "Interpretable Explanations of Black Boxes by Meaningful Perturbation" do?

I do not understand the purpose of the $\lambda$ parameter in equation 3 of the paper Interpretable Explanations of Black Boxes by Meaningful Perturbation. $$m^{*}=\underset{m \in[0,1]^{\Lambda}}{\...
0
votes
2answers
49 views

Is pruning only applicable to convolutional neural networks?

This article talks about pruning in the context of convolutional neural networks: One of the first methods of pruning is pruning entire convolutional filters. Using an L1 norm of the weight of all ...
2
votes
1answer
87 views

What gets optimized in convolutional neural network?

In a convolutional neural network, the hyperparameters such as number of kernels and stride, kernel size, etc are determined. After some combination of convolutions, ReLU and pooling layer there is ...
1
vote
2answers
305 views

In logistic regression, why is the binary cross-entropy loss function convex?

I am studying logistic regression for binary classification. The loss function used is cross-entropy. For a given input $x$, if our model outputs $\hat{y}$ instead of $y$, the loss is given by $$\text{...
0
votes
0answers
20 views

Gradient of Scalar objective cannot be efficiently calculated?

Suppose we generate the vector output $y$ from model $h(x, \theta)$, with input $x$ and parameters $\theta$. Reverse mode differentiation says that we can calculate the gradient \begin{align*} \...
7
votes
1answer
270 views

When should you not use the bias in a layer?

I'm not really that experienced with deep learning, and I've been looking at research code (mostly PyTorch) for deep neural networks, specifically GANs, and, in many cases, I see the authors setting <...
0
votes
0answers
50 views

Any RL approaches for this 2D space optimization problem?

I have a list of rectangles, they are in a certain order in 2D at the beginning. The task is to move them to get the boundary (rectangular) of the minimal area. It's OK to push off the dotted border ...
0
votes
1answer
83 views

Is it possible to optimize a multi-variable function with a reinforcement learning method?

I want to use RL instead of genetic or any other evolutionary algorithm in order to find the best parameter for a function. Here is the problem: Given a function $$f(x,y,z, \text{data}),$$ where $x$, $...
1
vote
0answers
35 views

Optimizer that prevents parameters from oscillating

When we perform gradient descent, especially in an online setting where the training data is presented in a non-random order, a particular 1-dimensional parameter (such as an edge weight) may first ...
1
vote
0answers
34 views

What are some use cases of discrete optimization in Deep Learning?

When we talk of optimization, it usually boils down to gradient descent and its variants in the context of deep learning. However, I wonder if there are some works that use discrete optimization in ...
0
votes
0answers
40 views

What are the best optimizations I can add to my neural network?

I am making an artificial neural network from scratch (without nn libraries) in python. So, as you can guess, its extremely unoptimized and slow. For this neural ...
2
votes
1answer
216 views

How to avoid being stuck local optima in q-learning and q-network

When using the Bellman equation to update q-table or train q-network to fit greedy max values, the q-values very often get to the local optima and get stuck although randomization rate ($\epsilon$) ...
1
vote
1answer
83 views

How to deal with evolutionary/genetic fitness function that can have both negative and positive values?

I am optimising function that can have both positive and negative values in pretty much unknown ranges, might be -100, 30, 0.001, or 4000, or -0.4 and I wonder how I can transform these results so I ...
5
votes
0answers
71 views

Why are most commonly used activation functions continuous?

I have come to notice that the most commonly used activation functions are continuous. Is there any specific reason behind this? Results such as this paper have worked on training networks with ...
1
vote
0answers
55 views

How can I model this problem as optimization problem that can be solved with ACO?

I have the following homework problem, where I need to explain how to model a certain problem as an optimization problem and how I can solve it with ACO. You have 6 portable media players $(P_i, ...
2
votes
0answers
26 views

Is it possible to ensure the convergence when training a RNN weight on its SVD decomposition?

I'm reading the following paper in which the author seems to do 2 things interesting: The hidden-to-hidden weight matrix of the RNN is SVD decomposed and train separately. Each orthogonal part of the ...
0
votes
1answer
91 views

How to make input variable as trainable parameter in a neural network?

I am working on an optimization problem. First, I have done forward training to work the network as a surrogate model, then I freeze the output and I want to find an optimal value of input for a given ...
1
vote
0answers
28 views

How to derive compact convex set K and its diameter D to program Accelegrad algorithm in practice?

Given the original paper (https://arxiv.org/pdf/1809.02864.pdf), I would like to implement the Accelegrad algorithm for which I report the pseudocode of the paper: In the pseudocode, the authors ...
2
votes
0answers
46 views

How does the support vector machine constraint imply that sample selection bias will not systematically affect the output of the optimisation?

I am currently studying the paper Learning and Evaluating Classifiers under Sample Selection Bias by Bianca Zadrozny. In section 3.4. Support vector machines, the author says the following: 3.4. ...
0
votes
0answers
75 views

What is the best algorithm for optimizing profit, rather than making predictions?

I am new to machine learning, so I am not sure which algorithms to look at for my business problem. Most of what I am seeing in tools like KNIME are geared toward making a prediction/classification, ...
1
vote
1answer
71 views

In the update rule of RMSprop, do we divide by a matrix?

I've been trying to understand RMSprop for a long time, but there's something that keeps eluding me. Here is a screenshot from this video by Andrew Ng. From the element-wise comment, from what I ...
2
votes
1answer
205 views

How are these equations of SGD with momentum equivalent?

I know this question may be so silly, but I can not prove it. In Stanford slide (page 17), they define the formula of SGD with momentum like this: $$ v_{t}=\rho v_{t-1}+\nabla f(x_{t-1}) \\ x_{t}=x_{...
1
vote
1answer
51 views

Why is Adam trapped in bad/suspicious local optima after the first few updates?

In the paper On the Variance of the Adaptive Learning Rate and Beyond, in section 2, the authors write To further analyze this phenomenon, we visualize the histogram of the absolute value of ...
2
votes
1answer
242 views

Why is second-order backpropagation useful?

Raul Rojas's book on Neural Networks dedicates section 8.4.3 to explaining how to do second-order backpropagation, that is, computing the Hessian of the error function with respect to two weights at a ...