Questions tagged [optimization]

For questions about implementing and improving optimization algorithms used in creating AI programs, or optimization in general.

Filter by
Sorted by
Tagged with
1
vote
0answers
22 views

How do you decide that you have tested enough hyper-parameter combinations for a specific neural network architecture?

How do you decide that you have tested enough hyper-parameter combinations for a specific neural network architecture to discard it and move on to a new model? Do you have a structured (generic) ...
0
votes
1answer
31 views

Compare the efficiency of a trained ML model with a non-learning-based method for solving the same problem

If a certain task T is solved by a non-learning-based method A (let's say, an optimization-based approach). We now train a machine learning model B (let's say a neural network) on the same task. What ...
0
votes
0answers
23 views

How do I solve a minimization problem with Q-learning learning?

I am trying to learn reinforcement learning by myself and so I have a lot of doubts. In particular, I am investigating how to use Q-learning in order to solve minimization problems. For example, ...
0
votes
1answer
26 views

Which class of functions are quite complicated in deep learning?

Deep learning is a field in which we need neural networks that are deep enough to carry on our task. The important fucntions in deep neural networks can be classified in to three classes: activation ...
1
vote
1answer
25 views

Do gradient-based algorithms deal with the flat regions with desired points?

I am studying a chapter named Numerical Computation of a deep learning book. Afaik, it does not deal with flat regions with desired points. For example, let us consider a function whose local/global ...
0
votes
1answer
45 views

How many directions of gradients exist for a function in higher dimensional space?

Gradient is used in optimization algorithms. Based on the values of gradient, we generally update weights of a neural network. It is known that gradient have a direction and the direction opposite to ...
1
vote
0answers
30 views

Different learning rates for different layers

I wonder, whether there are heuristic rules for optimal selection of learning rates for different layers. I expect, that there is no general recipe, but probably there are some some choices that maybe ...
0
votes
2answers
33 views

Reason for relaxing limit in derivative in this context?

Consider the following paragraph from NUMERICAL COMPUTATION of the deep learning book.. Suppose we have a function $y = f(x)$, where both $x$ and $y$ are real numbers. The derivative of this function ...
0
votes
0answers
12 views

What is meant by non-convergent limit cycles?

Limit cycle is a closed curve that is isolated i.e., no other closed curve near to it. You can read the following paragraph from here If there is (such) a closed curve, the nearby trajectories must ...
1
vote
1answer
19 views

Cover a surface with smaller predefined objects

I'm trying to make a program that takes a surface designed by the user, and different 3D geometries from a dataset as inputs and gives a good approximation of the surface using only the objects found ...
3
votes
1answer
56 views

What are the necessary mathematical properties to be a loss function in gradient based optimizations?

Loss functions are used in training neural networks. I am interested in knowing the mathematical properties that are necessary for a loss function to participate in gradient descent optimization. I ...
1
vote
0answers
14 views

What does the lambda parameter in the paper "Interpretable Explanations of Black Boxes by Meaningful Perturbation" do?

I do not understand the purpose of the $\lambda$ parameter in equation 3 of the paper Interpretable Explanations of Black Boxes by Meaningful Perturbation. $$m^{*}=\underset{m \in[0,1]^{\Lambda}}{\...
1
vote
2answers
35 views

Is "Pruning" only applicable to CNNs?

What Is Neural Network Pruning And Why Is It Important Today? The above article only talks about Convolutional Neural Networks: One of the first methods of pruning is pruning entire convolutional ...
2
votes
1answer
64 views

What gets optimized in convolutional neural network?

In a convolutional neural network, the hyperparameters such as number of kernels and stride, kernel size, etc are determined. After some combination of convolutions, ReLU and pooling layer there is ...
2
votes
1answer
108 views

In logistic regression, why is the binary cross-entropy loss function convex?

I am studying logistic regression for binary classification. The loss function used is cross-entropy. For a given input $x$, if our model outputs $\hat{y}$ instead of $y$, the loss is given by $$\text{...
1
vote
0answers
19 views

Gradient of Scalar objective cannot be efficiently calculated?

Suppose we generate the vector output $y$ from model $h(x, \theta)$, with input $x$ and parameters $\theta$. Reverse mode differentiation says that we can calculate the gradient \begin{align*} \...
0
votes
0answers
18 views

How to improve Face matching time

I'm working on a project that aims to detect each person's face while entering to a public space and store entering time and the person's image (array format) in Elasticsearch, and then detect each ...
7
votes
1answer
151 views

When should you not use the bias in a layer?

I'm not really that experienced with deep learning, and I've been looking at research code (mostly PyTorch) for deep neural networks, specifically GANs, and, in many cases, I see the authors setting <...
0
votes
0answers
37 views

Any RL approaches for this 2D space optimisation problem?

I have a list of rectangles, they are in certain order in 2D at the beginning. The task is to move them to get the boundary (rectangular) of the minimal area. It's OK to push off the dotted border as ...
0
votes
1answer
64 views

Use Reinforcement Learning instead of genetic algorithm for optimization

I want to use RL instead of genetic or any other evolutionary algorithm in order to find the best parameter for a function. Here is the problem: Given a function $$f(x,y,z,data)$$ x,y and z are some ...
1
vote
0answers
34 views

Optimizer that prevents parameters from oscillating

When we perform gradient descent, especially in an online setting where the training data is presented in a non-random order, a particular 1-dimensional parameter (such as an edge weight) may first ...
1
vote
0answers
29 views

What are some use cases of discrete optimization in Deep Learning?

When we talk of optimization, it usually boils down to gradient descent and its variants in the context of deep learning. However, I wonder if there are some works that use discrete optimization in ...
0
votes
0answers
36 views

What are the best optimizations I can add to my neural network?

I am making an artificial neural network from scratch (without nn libraries) in python. So, as you can guess, its extremely unoptimized and slow. For this neural ...
2
votes
1answer
114 views

How to avoid being stuck local optima in q-learning and q-network

When using Bellman equation to update q-table or train q-network to fit to greedy max values, the q-values very often get to the local optima and get stuck although randomisation rate ($\epsilon$) ...
1
vote
1answer
61 views

How to deal with evolutionary/genetic fitness function that can have both negative and positive values?

I am optimising function that can have both positive and negative values in pretty much unknown ranges, might be -100, 30, 0.001, or 4000, or -0.4 and I wonder how I can transform these results so I ...
4
votes
0answers
59 views

Why are most commonly used activation functions continuous?

I have come to notice that the most commonly used activation functions are continuous. Is there any specific reason behind this? Results such as this paper have worked on training networks with ...
1
vote
0answers
46 views

How can I model this problem as optimization problem that can be solved with ACO?

I have the following homework problem, where I need to explain how to model a certain problem as an optimization problem and how I can solve it with ACO. You have 6 portable media players $(P_i, ...
2
votes
0answers
23 views

Is it possible to ensure the convergence when training a RNN weight on its SVD decomposition?

I'm reading the following paper in which the author seems to do 2 things interesting: The hidden-to-hidden weight matrix of the RNN is SVD decomposed and train separately. Each orthogonal part of the ...
0
votes
1answer
56 views

How to make input variable as trainable parameter in a neural network?

I am working on an optimization problem. First, I have done forward training to work the network as a surrogate model, then I freeze the output and I want to find an optimal value of input for a given ...
1
vote
0answers
28 views

How to derive compact convex set K and its diameter D to program Accelegrad algorithm in practice?

Given the original paper (https://arxiv.org/pdf/1809.02864.pdf), I would like to implement the Accelegrad algorithm for which I report the pseudocode of the paper: In the pseudocode, the authors ...
0
votes
0answers
15 views

What do you keep in mind while defining a perturbation function for ILS algorithm?

I've been having troubles in defining the perturb function for the problem I am working on. I know that this function should help you in "jumping" to another "hill", but not so far ...
0
votes
0answers
20 views

What to do when Iterated Local Search (ILS) method keeps making the solution worse?

I've been trying to solve an optimization problem with ITERATED LOCAL SEARCH (ILS) method. I've generated the initial solution, then followed the steps of ILS. However, even after running the ...
2
votes
0answers
46 views

How does the support vector machine constraint imply that sample selection bias will not systematically affect the output of the optimisation?

I am currently studying the paper Learning and Evaluating Classifiers under Sample Selection Bias by Bianca Zadrozny. In section 3.4. Support vector machines, the author says the following: 3.4. ...
0
votes
0answers
68 views

What is the best algorithm for optimizing profit, rather than making predictions?

I am new to machine learning, so I am not sure which algorithms to look at for my business problem. Most of what I am seeing in tools like KNIME are geared toward making a prediction/classification, ...
0
votes
1answer
55 views

RMSprop equation - dividing by a matrix?

I've been trying to understand RMSprop for a long time, but there's something that keeps eluding me. $dW$ and $db$ are matrices (that's what I understand from the element-wise comment), so that must ...
1
vote
1answer
132 views

How are these equations of SGD with momentum equivalent?

I know this question may be so silly, but I can not prove it. In Stanford slide (page 17), they define the formula of SGD with momentum like this: $$ v_{t}=\rho v_{t-1}+\nabla f(x_{t-1}) \\ x_{t}=x_{...
1
vote
1answer
44 views

Why is Adam trapped in bad/suspicious local optima after the first few updates?

In the paper On the Variance of the Adaptive Learning Rate and Beyond, in section 2, the authors write To further analyze this phenomenon, we visualize the histogram of the absolute value of ...
2
votes
1answer
176 views

Why is second-order backpropagation useful?

Raul Rojas's book on Neural Networks dedicates section 8.4.3 to explaining how to do second-order backpropagation, that is, computing the Hessian of the error function with respect to two weights at a ...
1
vote
1answer
47 views

What are the conceptual differences between regularisation and optimisation in deep neural nets?

I'm trying to wrap my mind around the concepts of regularisation and optimisation in neural nets, especially around their differences. In my current understanding, regularisation is intended to tackle ...
1
vote
0answers
53 views

What is the difference between exploitation and exploration in the context of optimization?

In the paper Moth-flame optimization algorithm: A novel nature-inspired heuristic paradigm (2015, published in Knowledge-Based Systems) The test functions are divided to three groups: unimodal, multi-...
0
votes
0answers
19 views

How to afine the extremity values in regression prediction with Keras?

I made a stack of bidirectional LSTM layers following by Dense layers (with swish activation functions) in order to predict a continuous value between 0 and 2. I compiled the model with ...
8
votes
2answers
249 views

Why is the perceptron criterion function differentiable?

I'm reading chapter one of the book called Neural Networks and Deep Learning from Aggarwal. In section 1.2.1.1 of the book, I'm learning about the perceptron. One thing that book says is, if we use ...
2
votes
0answers
34 views

How are the lower and upper bound values of the moths determined in the Moth-Flame Optimization algorithm?

I am currently implementing the Moth-Flame Optimization (MFO) Algorithm, based on the paper: Moth-Flame Optimization Algorithm: A Novel Nature-inspired Heuristic Paradigm. To calculate the values of ...
3
votes
1answer
46 views

Measuring novel configuration of points

I am trying to implement Novelty search; I understand why it can work better than the standard Genetic Algorithm based solution which just rewards according to the objective. I am working on a problem ...
1
vote
0answers
15 views

Root finding in Deep Equilibrium Models

In the Deep Equilibrium Model the neural network can be seen as "infinitely deep". Training learns a nonlinear function as usual. But there is no forward propagation of input data through ...
1
vote
0answers
27 views

Why scaling down the parameter many times during training will help the learning speed be the same for all weights in Progressive GAN?

The title is one of the special things in Progressive GAN, a paper of the NVIDIA team. By using this method, they introduced that Our approach ensures that the dynamic range, and thus the learning ...
3
votes
1answer
153 views

Why is the learning rate generally beneath 1?

In all examples I've ever seen, the learning rate of an optimisation method is always less than $1$. However, I've never found an explanation as to why this is. In addition to that, there are some ...
2
votes
2answers
124 views

How does PCA work when we reduce the original space to 2 or higher-dimensional space?

How does PCA work when we reduce the original space to a 2 or higher-dimensional space? I understand the case when we reduce the dimensionality to $1$, but not this case. $$\begin{array}{ll} \text{...
1
vote
2answers
62 views

Why does Simulated Annealing not take worse solution if the energy difference becomes higher?

In Simulated Annealing, a worse solution is accepted with this probability: $$p=e^{-\frac{E(y)-E(x)}{kT}}.$$ If that understanding is correct: Why is this probability function used? This means that, ...
1
vote
0answers
24 views

Which method of tree searching should be used for this board game?

Suppose the following properties of a board game: High branching factor in the beginning of the game (~500) which slowly tends towards 0 at the end of the game Evaluation of the any given board ...