Questions tagged [optimization]

For questions about implementing and improving optimization algorithms used in creating AI programs, or optimization in general.

80 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
4 votes
0 answers
100 views

Why are most commonly used activation functions continuous?

I have come to notice that the most commonly used activation functions are continuous. Is there any specific reason behind this? Results such as this paper have worked on training networks with ...
user avatar
  • 495
4 votes
0 answers
208 views

How does SGD escape local minima?

SGD is able to jump out of local minima that would otherwise trap BGD I don't really understand the above statement. Could someone please provide a mathematical explanation for why SGD (Stochastic ...
user avatar
4 votes
1 answer
598 views

When training a CNN, what are the hyperparameters to tune first?

I am training a convolutional neural network for object detection. Apart from the learning rate, what are the other hyperparameters that I should tune? And in what order of importance? Besides, I read ...
user avatar
  • 41
4 votes
1 answer
49 views

How to use MOPSO to align characters vertically?

I need to efficiently align characters vertically using Multi Objective PSO. Alignment is achieved by adding spaces in between a given set of characters. ...
user avatar
  • 243
3 votes
0 answers
98 views

Does Retina-net's focal loss accomplish its goal?

Taking out the weighting factor we can define focal loss as $$FL(p) = -(1-p)^\gamma log(p) $$ Where $p$ is the target probability. The idea being that single stage object detectors have a huge ...
user avatar
  • 2,249
2 votes
0 answers
23 views

How to create a loss function that penalizes duplicate indices in the output tensor?

We're working on a sequence-to-sequence problem using pytorch, and are using cross-entropy to calculate the loss when comparing the output sequence to the target sequence. This works fine and ...
user avatar
  • 121
2 votes
1 answer
93 views

Are Genetic Algorithms suitable for a problem with a non-unique optimal solution?

I was wondering if a genetic algorithm is useful if the optimization problem has several optimal solutions. My thought was that I should not use it since when combining two members of a population who ...
user avatar
  • 121
2 votes
0 answers
28 views

Is it possible to ensure the convergence when training a RNN weight on its SVD decomposition?

I'm reading the following paper in which the author seems to do 2 things interesting: The hidden-to-hidden weight matrix of the RNN is SVD decomposed and train separately. Each orthogonal part of the ...
user avatar
2 votes
0 answers
46 views

How does the support vector machine constraint imply that sample selection bias will not systematically affect the output of the optimisation?

I am currently studying the paper Learning and Evaluating Classifiers under Sample Selection Bias by Bianca Zadrozny. In section 3.4. Support vector machines, the author says the following: 3.4. ...
user avatar
2 votes
0 answers
38 views

How are the lower and upper bound values of the moths determined in the Moth-Flame Optimization algorithm?

I am currently implementing the Moth-Flame Optimization (MFO) Algorithm, based on the paper: Moth-Flame Optimization Algorithm: A Novel Nature-inspired Heuristic Paradigm. To calculate the values of ...
user avatar
2 votes
1 answer
172 views

Continuous state and continuous action Markov decision process time complexity estimate: backward induction VS policy gradient method (RL)

Model Description: Model based(assume known of the entire model) Markov decision process. Time($t$): Finite horizon discrete time with discounting factor State($x_t$): Continuous multi-dimensional ...
user avatar
2 votes
0 answers
38 views

Is logistic regression used for unconstrained or constrained optimisation problems?

Is logistic regression used for unconstrained or constrained optimization problems, and why?
user avatar
  • 21
2 votes
0 answers
115 views

How can I assign agents to tasks based on time and affinity?

I am working on an assignment problem. Consider $K$ agents $A_1, \dots A_K$ and $N$ tasks $T_1, \dots T_N$. Each task has a certain time $t(T_i)$ to be completed and each agent has a matching (or ...
user avatar
2 votes
0 answers
55 views

Is a neural network the correct approach to optimising a fitness function in a genetic algorithm?

I've written an application to help players pick the optimal heroes during the draft phase of the Heroes of the Storm MOBA. It can be daunting to pick from 80+ characters that have synergies/counters ...
user avatar
2 votes
0 answers
66 views

Any guidance on learning rate / batch size for noisy data (high Bayes error rate)?

Is there any guidance available for training on very noisy data, when Bayes error rate (lowest possible error rate for any classifier) is high? For example, I wonder if deliberately (not due to memory ...
user avatar
  • 151
2 votes
0 answers
127 views

Is a very powerful oracle sufficient to trigger the AI singularity?

Lets say we have a oracle $S$ that, given any function $F$ and desired output $y$, can find an input $x$ that causes $F$ to output $y$ if it exists, or otherwise returns nil. I.e.: $$S(F, y) = x \...
user avatar
  • 259
2 votes
0 answers
90 views

Which features and algorithm could optimize this air-conditioner problem?

Imagine we have 2 air conditioner systems (AA) and 2 "free cooling" systems which mix external and internal air (FC) in a closed box which always tends to warm up. For each system, we have to find ...
user avatar
  • 236
1 vote
0 answers
31 views

Do NNs suffer from lack of efficiency in network structure and suggesting training parameters?

I am working on dynamical systems using Optimal Control theory and trying to find the connection between this field and Machine Learning. Consider a simple 2-layer Neural Network (NN) where the ...
user avatar
1 vote
0 answers
28 views

Structured policies in dynamic programming: solving a toy example

I am trying to solve a dynamic programming toy example. Here is the prompt: imagine you arrive in a new city for $N$ days and every night need to pick a restaurant to get dinner at. The qualities of ...
user avatar
1 vote
0 answers
43 views

Best algorithm for the Word Ladder puzzle

What would be the best performing algorithm to solve the Word Ladder problem, in terms of guaranteed finding of the shortest solution in the shortest possible time? Is it BFS, DFS, A*, IDA* or another ...
user avatar
1 vote
0 answers
47 views

Are Bayesian Optimization Methods Better Suited Noisy Optimization Problems?

We know that in many applied contexts (e.g. Machine Learning, Loss Functions for Neural Networks), the functions we are trying to optimize are "noisy" by definition (unlike in the classical ...
user avatar
1 vote
0 answers
148 views

Why does the schema theorem of genetic algorithms hold?

I have been reading about the Schema Theorem - one of the first theorems from the field of evolutionary computing and genetic algorithms, largely responsible for justifying the use of genetic ...
user avatar
1 vote
1 answer
53 views

Does reaching the global optima guarantee good performance in a task?

It is to my understanding that, in deep learning, we are essentially trying to minimize the loss function that we have defined and reach its global optima through some form of optimization technique. ...
user avatar
  • 123
1 vote
0 answers
24 views

a loss for binary step function data

I have some data with ground truth that looks like a binary step function, where part of it is 0 and part is one. An example for the GT can be like ...
user avatar
  • 111
1 vote
0 answers
37 views

Multi-armed Bandit in optimization on graph edges selection

I have the problem, which I described below. I wonder if there exists a class of multi-armed bandit approaches that is related to it. I am working on computer networking optimization. In the simplest ...
user avatar
  • 21
1 vote
0 answers
49 views

Why are optimization algorithms for deep learning so simple?

From my knowledge, the most used optimizer in practice is Adam, which in essence is just mini-batch gradient descent with momentum to combat getting stuck in saddle points and with some damping to ...
user avatar
1 vote
0 answers
28 views

How do you decide that you have tested enough hyper-parameter combinations for a specific neural network architecture?

How do you decide that you have tested enough hyper-parameter combinations for a specific neural network architecture to discard it and move on to a new model? Do you have a structured (generic) ...
user avatar
  • 11
1 vote
0 answers
17 views

What does the lambda parameter in the paper "Interpretable Explanations of Black Boxes by Meaningful Perturbation" do?

I do not understand the purpose of the $\lambda$ parameter in equation 3 of the paper Interpretable Explanations of Black Boxes by Meaningful Perturbation. $$m^{*}=\underset{m \in[0,1]^{\Lambda}}{\...
user avatar
1 vote
0 answers
36 views

Optimizer that prevents parameters from oscillating

When we perform gradient descent, especially in an online setting where the training data is presented in a non-random order, a particular 1-dimensional parameter (such as an edge weight) may first ...
user avatar
  • 141
1 vote
0 answers
35 views

What are some use cases of discrete optimization in Deep Learning?

When we talk of optimization, it usually boils down to gradient descent and its variants in the context of deep learning. However, I wonder if there are some works that use discrete optimization in ...
user avatar
1 vote
0 answers
31 views

How to derive compact convex set K and its diameter D to program Accelegrad algorithm in practice?

Given the original paper (https://arxiv.org/pdf/1809.02864.pdf), I would like to implement the Accelegrad algorithm for which I report the pseudocode of the paper: In the pseudocode, the authors ...
user avatar
  • 149
1 vote
1 answer
60 views

Why is Adam trapped in bad/suspicious local optima after the first few updates?

In the paper On the Variance of the Adaptive Learning Rate and Beyond, in section 2, the authors write To further analyze this phenomenon, we visualize the histogram of the absolute value of ...
user avatar
1 vote
0 answers
64 views

What is the difference between exploitation and exploration in the context of optimization?

In the paper Moth-flame optimization algorithm: A novel nature-inspired heuristic paradigm (2015, published in Knowledge-Based Systems) The test functions are divided to three groups: unimodal, multi-...
user avatar
1 vote
0 answers
17 views

Root finding in Deep Equilibrium Models

In the Deep Equilibrium Model the neural network can be seen as "infinitely deep". Training learns a nonlinear function as usual. But there is no forward propagation of input data through ...
user avatar
  • 111
1 vote
0 answers
27 views

Why scaling down the parameter many times during training will help the learning speed be the same for all weights in Progressive GAN?

The title is one of the special things in Progressive GAN, a paper of the NVIDIA team. By using this method, they introduced that Our approach ensures that the dynamic range, and thus the learning ...
user avatar
  • 817
1 vote
0 answers
32 views

Which method of tree searching should be used for this board game?

Suppose the following properties of a board game: High branching factor in the beginning of the game (~500) which slowly tends towards 0 at the end of the game Evaluation of the any given board ...
user avatar
  • 111
1 vote
0 answers
38 views

How many training runs are needed to obtain a credible value for performance?

I'm trying to optimize a neural network. For that, I'm changing parameters like the batch size, learning rate, weight initialization, etc. A neural network is not a deterministic algorithm, so, in ...
user avatar
1 vote
0 answers
49 views

Simplifying Log Loss

I am reading through a paper (https://www.mitpressjournals.org/doi/pdf/10.1162/0891201053630273) where they describe logloss as a ranking function and can be simplified to the margin of the training ...
user avatar
  • 205
1 vote
3 answers
140 views

How to improve neural network training against a large data set of points with varying magnitude

I am currently using TensorFlow and have simply been trying to train a neural network directly against a large continuous data set, e.g. $y = [0.014, 1.545, 10.232, 0.948, ...]$ corresponding to ...
user avatar
1 vote
0 answers
44 views

Are there optimizers that schedule their learning rate, momentum etc. autonomously?

I'm aware there are some optimizer such as Adam that adjust the learning rate for each dimension during training. However, afaik, the maximum learning rate they can have is still determined by the ...
user avatar
1 vote
0 answers
21 views

Optimizer effects on neural network with two outputs

I'm confused about the following issue. Let assume that we have a neural network that takes one input and two outputs. I try to visualize my model like as follows: ...
user avatar
  • 11
1 vote
0 answers
34 views

Imposing contraints on sequence of image classifications

Are there example implementations of networks that apply constraints across sequences of image classifications where class labels are ordinal numbers? For example, to cause the output of a CNN to ...
user avatar
1 vote
0 answers
275 views

How does the automated temperature adjustment step work in Soft Actor-Critic?

In section 5 of the paper Soft Actor-Critic Algorithms and Applications, it is proposed an optimization problem to obtain an optimal temperature parameter $\alpha^*_t$. First, one uses the original ...
user avatar
1 vote
0 answers
68 views

Is convergence to a local minima more likely with transfer learning?

While doing transfer learning where my two problems are face-generation and car-generation is it likely that, if I use the weights of one problem as the initialization of the weights for the other ...
user avatar
  • 1,359
1 vote
0 answers
227 views

Could the Jensen-Shannon divergence and Kullback-Leibler divergence be used as loss functions of non-generation problems?

If I understand correctly, the KL divergence is a measure of information loss between a ground truth distribution $P$ and a predicted distribution $Q$, and the Jensen-Shannon divergence is the mean of ...
user avatar
  • 1,359
1 vote
0 answers
26 views

Estimating Baselines using ALS

I am trying to figure out how ALS works when minimizing the following formula: $\\ \\$ $\text{min}_{\lbrace b_u,b_i \rbrace} \sum_{(u,i)\in \mathcal{K}} (r_{ui} - \bar{r} - b_u - b_i )^2 + \lambda_{...
user avatar
1 vote
1 answer
203 views

How to properly optimize shared network between actor and critic?

I'm building an actor-critic reinforcment learning algorithm to solve environments. I want to use a single encoder to find representation of my environment. When I share the encoder with the actor ...
user avatar
  • 181
1 vote
0 answers
21 views

What is the purpose of the new neurons in the constrained neural network?

I would like to train a constrained neural network. I found a paper on this: https://papers.nips.cc/paper/4-constrained-differential-optimization.pdf. However, I don't really understand how to change ...
user avatar
  • 41
1 vote
0 answers
22 views

Feature visualization on neural networks which are not for classification

Feature visualization allows to better understand neural networks by generating images that maximize the activation of a specific neuron, and therefore understand what are the abstract features that ...
user avatar
  • 269
1 vote
0 answers
34 views

Classical Internet routing vs. Swarm routing (such as Ant routing)?

Is it possible to mention the drawbacks/advantages of Swarm routing (such as Ant routing etc) in comparison with classical routing algorithms in communication networks in a general view? In other ...
user avatar