Questions tagged [optimization]

For questions about implementing and improving optimization algorithms used in creating AI programs, or optimization in general.

59 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
7
votes
0answers
76 views

Given a list of integers $\{c_1, \dots, c_N \}$, how do I find an integer $D$ that minimizes the sum of remainders $\sum_i c_i \text{ mod } D$?

I have a set of fixed integers $S = \{c_1, \dots, c_N \}$. I want to find a single integer $D$, greater than a certain threshold $T$, i.e. $D > T \geq 0$, that divides each $c_i$ and leaves ...
4
votes
1answer
41 views

How to use MOPSO to align characters vertically?

I need to efficiently align characters vertically using Multi Objective PSO. Alignment is achieved by adding spaces in between a given set of characters. ...
3
votes
0answers
81 views

How does SGD escape local minima?

SGD is able to jump out of local minima that would otherwise trap BGD I don't really understand the above statement. Could someone please provide a mathematical explanation for why SGD (Stochastic ...
3
votes
0answers
75 views

Does Retina-net's focal loss accomplish its goal?

Taking out the weighting factor we can define focal loss as $$FL(p) = -(1-p)^\gamma log(p) $$ Where $p$ is the target probability. The idea being that single stage object detectors have a huge ...
3
votes
0answers
88 views

Which features and algorithm could optimize this air-conditioner problem?

Imagine we have 2 air conditioner systems (AA) and 2 "free cooling" systems which mix external and internal air (FC) in a closed box which always tends to warm up. For each system, we have to find ...
3
votes
0answers
37 views

Knapsack of mixture with constraints

I'm trying to find the optimized mixture for a specific set of substances. Each of those substances have characteristics that I want to optimize in the mixture (some characteristics I want to minimize ...
2
votes
0answers
12 views

Why are most commonly used activation functions continuous?

I have come to notice that the most commonly used activation functions are continuous. Is there any specific reason behind this? Results such as this paper have worked on training networks with ...
2
votes
0answers
18 views

Is it possible to ensure the convergence when training a RNN weight on its SVD decomposition?

I'm reading the following paper in which the author seems to do 2 things interesting: The hidden-to-hidden weight matrix of the RNN is SVD decomposed and train separately. Each orthogonal part of the ...
2
votes
0answers
42 views

How does the support vector machine constraint imply that sample selection bias will not systematically affect the output of the optimisation?

I am currently studying the paper Learning and Evaluating Classifiers under Sample Selection Bias by Bianca Zadrozny. In section 3.4. Support vector machines, the author says the following: 3.4. ...
2
votes
0answers
30 views

How are the lower and upper bound values of the moths determined in the Moth-Flame Optimization algorithm?

I am currently implementing the Moth-Flame Optimization (MFO) Algorithm, based on the paper: Moth-Flame Optimization Algorithm: A Novel Nature-inspired Heuristic Paradigm. To calculate the values of ...
2
votes
1answer
108 views

When training a CNN, what are the hyperparameters to tune first?

I am training a convolutional neural network for object detection. Apart from the learning rate, what are the other hyperparameters that I should tune? And in what order of importance? Besides, I read ...
2
votes
0answers
36 views

Is logistic regression used for unconstrained or constrained optimisation problems?

Is logistic regression used for unconstrained or constrained optimization problems, and why?
2
votes
0answers
114 views

How can I assign agents to tasks based on time and affinity?

I am working on an assignment problem. Consider $K$ agents $A_1, \dots A_K$ and $N$ tasks $T_1, \dots T_N$. Each task has a certain time $t(T_i)$ to be completed and each agent has a matching (or ...
2
votes
0answers
55 views

Is a neural network the correct approach to optimising a fitness function in a genetic algorithm?

I've written an application to help players pick the optimal heroes during the draft phase of the Heroes of the Storm MOBA. It can be daunting to pick from 80+ characters that have synergies/counters ...
2
votes
0answers
98 views

Is a very powerful oracle sufficient to trigger the AI singularity?

Lets say we have a oracle $S$ that, given any function $F$ and desired output $y$, can find an input $x$ that causes $F$ to output $y$ if it exists, or otherwise returns nil. I.e.: $$S(F, y) = x \...
2
votes
0answers
20 views

What does it essentially mean if the neural network has convex error surface?

Suppose if I am building a Linear Regression model with one fully connected layer and a sigmoid with minimizing mean squared error as objective. Why would the error surface be convex? Does finding ...
1
vote
0answers
30 views

How can I model this problem as optimization problem that can be solved with ACO?

I have the following homework problem, where I need to explain how to model a certain problem as an optimization problem and how I can solve it with ACO. You have 6 portable media players $(P_i, ...
1
vote
0answers
26 views

How to derive compact convex set K and its diameter D to program Accelegrad algorithm in practice?

Given the original paper (https://arxiv.org/pdf/1809.02864.pdf), I would like to implement the Accelegrad algorithm for which I report the pseudocode of the paper: In the pseudocode, the authors ...
1
vote
0answers
12 views

What do you keep in mind while defining a perturbation function for ILS algorithm?

I've been having troubles in defining the perturb function for the problem I am working on. I know that this function should help you in "jumping" to another "hill", but not so far ...
1
vote
0answers
19 views

What to do when Iterated Local Search (ILS) method keeps making the solution worse?

I've been trying to solve an optimization problem with ITERATED LOCAL SEARCH (ILS) method. I've generated the initial solution, then followed the steps of ILS. However, even after running the ...
1
vote
1answer
27 views

Why is Adam trapped in bad/suspicious local optima after the first few updates?

In the paper On the Variance of the Adaptive Learning Rate and Beyond, in section 2, the authors write To further analyze this phenomenon, we visualize the histogram of the absolute value of ...
1
vote
0answers
44 views

What is the difference between exploitation and exploration in the context of optimization?

In the paper Moth-flame optimization algorithm: A novel nature-inspired heuristic paradigm (2015, published in Knowledge-Based Systems) The test functions are divided to three groups: unimodal, multi-...
1
vote
0answers
13 views

Root finding in Deep Equilibrium Models

In the Deep Equilibrium Model the neural network can be seen as "infinitely deep". Training learns a nonlinear function as usual. But there is no forward propagation of input data through ...
1
vote
0answers
27 views

Why scaling down the parameter many times during training will help the learning speed be the same for all weights in Progressive GAN?

The title is one of the special things in Progressive GAN, a paper of the NVIDIA team. By using this method, they introduced that Our approach ensures that the dynamic range, and thus the learning ...
1
vote
0answers
19 views

Which method of tree searching should be used for this board game?

Suppose the following properties of a board game: High branching factor in the beginning of the game (~500) which slowly tends towards 0 at the end of the game Evaluation of the any given board ...
1
vote
1answer
59 views

Continuous state and continuous action Markov decision process time complexity estimate: backward induction VS policy gradient method (RL)

Model Description: Model based(assume known of the entire model) Markov decision process. Time($t$): Finite horizon discrete time with discounting factor State($x_t$): Continuous multi-dimensional ...
1
vote
0answers
36 views

How many training runs are needed to obtain a credible value for performance?

I'm trying to optimize a neural network. For that, I'm changing parameters like the batch size, learning rate, weight initialization, etc. A neural network is not a deterministic algorithm, so, in ...
1
vote
0answers
43 views

Simplifying Log Loss

I am reading through a paper (https://www.mitpressjournals.org/doi/pdf/10.1162/0891201053630273) where they describe logloss as a ranking function and can be simplified to the margin of the training ...
1
vote
3answers
101 views

How to improve neural network training against a large data set of points with varying magnitude

I am currently using TensorFlow and have simply been trying to train a neural network directly against a large continuous data set, e.g. $y = [0.014, 1.545, 10.232, 0.948, ...]$ corresponding to ...
1
vote
0answers
37 views

Are there optimizers that schedule their learning rate, momentum etc. autonomously?

I'm aware there are some optimizer such as Adam that adjust the learning rate for each dimension during training. However, afaik, the maximum learning rate they can have is still determined by the ...
1
vote
0answers
20 views

Optimizer effects on neural network with two outputs

I'm confused about the following issue. Let assume that we have a neural network that takes one input and two outputs. I try to visualize my model like as follows: ...
1
vote
0answers
29 views

Imposing contraints on sequence of image classifications

Are there example implementations of networks that apply constraints across sequences of image classifications where class labels are ordinal numbers? For example, to cause the output of a CNN to ...
1
vote
0answers
143 views

How does the automated temperature adjustment step work in Soft Actor-Critic?

In section 5 of the paper Soft Actor-Critic Algorithms and Applications, it is proposed an optimization problem to obtain an optimal temperature parameter $\alpha^*_t$. First, one uses the original ...
1
vote
0answers
42 views

Is convergence to a local minima more likely with transfer learning?

While doing transfer learning where my two problems are face-generation and car-generation is it likely that, if I use the weights of one problem as the initialization of the weights for the other ...
1
vote
0answers
141 views

Could the Jensen-Shannon divergence and Kullback-Leibler divergence be used as loss functions of non-generation problems?

If I understand correctly, the KL divergence is a measure of information loss between a ground truth distribution $P$ and a predicted distribution $Q$, and the Jensen-Shannon divergence is the mean of ...
1
vote
0answers
18 views

Estimating Baselines using ALS

I am trying to figure out how ALS works when minimizing the following formula: $\\ \\$ $\text{min}_{\lbrace b_u,b_i \rbrace} \sum_{(u,i)\in \mathcal{K}} (r_{ui} - \bar{r} - b_u - b_i )^2 + \lambda_{...
1
vote
1answer
58 views

How to properly optimize shared network between actor and critic?

I'm building an actor-critic reinforcment learning algorithm to solve environments. I want to use a single encoder to find representation of my environment. When I share the encoder with the actor ...
1
vote
0answers
20 views

What is the purpose of the new neurons in the constrained neural network?

I would like to train a constrained neural network. I found a paper on this: https://papers.nips.cc/paper/4-constrained-differential-optimization.pdf. However, I don't really understand how to change ...
1
vote
0answers
31 views

Any guidance on learning rate / batch size for noisy data (high Bayes error rate)?

Is there any guidance available for training on very noisy data, when Bayes error rate (lowest possible error rate for any classifier) is high? For example, I wonder if deliberately (not due to memory ...
1
vote
0answers
22 views

Feature visualization on neural networks which are not for classification

Feature visualization allows to better understand neural networks by generating images that maximize the activation of a specific neuron, and therefore understand what are the abstract features that ...
1
vote
0answers
32 views

Classical Internet routing vs. Swarm routing (such as Ant routing)?

Is it possible to mention the drawbacks/advantages of Swarm routing (such as Ant routing etc) in comparison with classical routing algorithms in communication networks in a general view? In other ...
1
vote
1answer
899 views

What are the advantages and disadvantages of using LISP for constraint satisfaction in 3D space

We are currently working on developing a 3D modeling software that allows designers to set spatial constraints to models. The computer then should generate a 3D mesh conforming to these constraints. ...
1
vote
0answers
50 views

Optimization step in Apprenticeship Learning via Inverse Reinforcement Learning

Why the optimization step of the algorithm a quadratic program? [See: Apprenticeship Learning via Inverse Reinforcement Learning; page 3] Isn't the objective function linear? Why don't we treat ...
1
vote
0answers
48 views

How to calculate Adaptive gradient?

In the FaceNet paper there mentions an gradient algorithm called 'AdaGrad'(Adaptive Gradient) referenced to this paper which has apparently been used to calculate the gradient of the Triplet Loss ...
1
vote
0answers
269 views

Application of Ai to task scheduling problems on heterogenous platforms

Let's say we have a cluster of 20-2000 heterogenous compute nodes. Consider for example the parallel solution of the helmholtz equation: Now we want to distribute the solution process and, to make ...
1
vote
0answers
76 views

Which algorithm would you use to solve a multiple producer-consumer problem with constraints?

I'm solving this problem similar to consumer-producer of materials (i.e. sand). This is the graph of the problem: Where Req (...
0
votes
1answer
30 views

How to make input variable as trainable parameter in a neural network?

I am working on an optimization problem. First, I have done forward training to work the network as a surrogate model, then I freeze the output and I want to find an optimal value of input for a given ...
0
votes
0answers
52 views

What is the best algorithm for optimizing profit, rather than making predictions?

I am new to machine learning, so I am not sure which algorithms to look at for my business problem. Most of what I am seeing in tools like KNIME are geared toward making a prediction/classification, ...
0
votes
1answer
39 views

RMSprop equation - dividing by a matrix?

I've been trying to understand RMSprop for a long time, but there's something that keeps eluding me. $dW$ and $db$ are matrices (that's what I understand from the element-wise comment), so that must ...
0
votes
0answers
18 views

How to afine the extremity values in regression prediction with Keras?

I made a stack of bidirectional LSTM layers following by Dense layers (with swish activation functions) in order to predict a continuous value between 0 and 2. I compiled the model with ...