# Questions tagged [optimization]

For questions about implementing and improving optimization algorithms used in creating AI programs, or optimization in general.

212 questions
Filter by
Sorted by
Tagged with
21 views

### Is GPU undervolting employed with powering AI models like chatgpt?

ChatGPT runs on thousands of GPUs that pull a lot of power and as a result, dissipate a lot of heat. This effectively makes the cost of powering the AI high in two aspects: power consumption Cooling ...
1 vote
28 views

### Applicability of Holland's Schema Theorem to Genetic Algorithms with Non-Binary Individual Representations

I'm currently working on a problem formulation that requires non-binary individual representations in a genetic algorithm (GA). I've been exploring Holland's Schema Theorem as a theoretical basis for ...
21 views

### Implementing momentum is causing calculation exceptions/errors

I am developing my own neural network in order to learn about how they work. I am implementing via C++ and the Eigen library (for matrix multiplication). I have a working implementation that seems to ...
1 vote
82 views

### Maximize a scoring function within the latent space of a generative model

Given a generative model, G, trained on a dataset D. This generative model can be either GAN or Diffusion based. Supposed each sample, x_i, generated by G, can be evaluated by a readily available ...
37 views

### Best way to generate fitness landscape when using higher dimensional data

I'm using a GA to find the best set of parameters to maximize a fitness function. I want to draw a fitness landscape to visualize the effectiveness of the algorithm. The fitness function, calculated ...
13 views

### What do I need to learn to tackle the following problem: make a program that optimizes decisions in the game PlateUp!

So, recently my friends and I have been hooked on a videogame called PlateUp!. The game is kind of a management game where the objective is to succesfully run a restaurant. The game can be roughly ...
13 views

### Finding an optimal action score function for Multi-Armed Bandit Problem

Considering a multi-armed bandit problem where there are : ...
1 vote
33 views

### How does Openai's CLIP avoids dimensional collapse?

According to this paper from FAIR : https://arxiv.org/abs/2110.09348 , contrastive learning methods suffer from the problem of dimensional collapse where "the embedding vectors end up spanning a ...
1 vote
56 views

### Cheap differentiable similarity metrics of vectors

I am looking to compute the similarity between a large set of vectors during neural network training - a process that is considerably expensive when choosing the wrong metric. So far, I am making use ...
1 vote
57 views

### Why are the non-linear activations in deep nets not learned?

Why can we not parametrize and learn the non-linear activations? For example, if we look at leaky ReLu which equals to $f(y)=y$ for $y>0$ and $f(y)=\alpha y$ for $y<0$, it seems that we can ...
34 views

### Behaviour of PPO/similar Algos under action penalties

I am currently experimenting with PPO in different environments. I am interested in learning policies that fulfill a certain goal while keeping a specific value low. Here's an example: Using PPO on a ...
43 views

### Poor convergence of a neural network, which implements NMF

I'd like to understand why this simple network fails to converge. The resulting MSE error is an order of 10^4 - 10^5 bigger than what could be achieved. The task is to do a non-negative matrix ...
1 vote
54 views

### Optimizing a blackbox function with binary states

I have a non-linear black box function, which inputs a vector(size=250) and outputs a scalar value; f(x) = value. The x variable is a vector of size 250 and has ...
37 views

### Very high dimensional optimization with large budget, requiring high quality solutions

What would be theoretically the best performing optimization algorithm(s) in this case? Very high dimensional problem: 250-500 parameters Goal is to obtain very high quality solutions, not just "...
1 vote
106 views

### How to estimate the gradient of an argmin loss

Suppose we have a neural network $f_\theta(x)$, where $x$ is the input and $\theta$ is the network's parameters. For each $\theta$, we can minimize $f_\theta(x)$ w.r.t. $x$ and obtain the minimum ...
70 views

### How to speed up my neural network?

I would like to train an LSTM-based variational autoencoder on a large dataset (37 million sentences). However, I have calculated that my training speed as of now is too slow (on Google Colab). I am ...
34 views

1 vote
31 views

### Additional Optimizations for Convolutional Models On Inferencing

I am aware of several ways to optimize a convolutional (or any) model after training to make inferencing quicker. I am currently implementing BatchNormalization Folding and removing Dropout layers ...
102 views

### What does it mean by Generalization? [closed]

Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learning What does it mean by Generalization in this article?
47 views

### Are Problems in AI Usually "Ill Posed"?

I was reading the following link (https://en.wikipedia.org/wiki/Well-posed_problem) on "Well Posed Problems". Supposedly, if a problem is "Well Posed", it must meet the following ...
1 vote
80 views

### Are Bayesian Optimization Methods Better Suited Noisy Optimization Problems?

We know that in many applied contexts (e.g. Machine Learning, Loss Functions for Neural Networks), the functions we are trying to optimize are "noisy" by definition (unlike in the classical ...
1 vote
1k views

### Effects of ReLU Activation on Convexity of Loss Functions

I have heard the following argument being made regarding Neural Networks: A Neural Network is a composition of several Activation Functions Sigmoid Activation Functions are Non-Convex Functions The ...
41 views

### What I Should Do to Reduce Solution Size for Simulated Annealing Algorithm?

I am trying to find the best solution for radar placement problem with using multi objective simulated annealing algorithm. So there is an area (in real map) and I want to put minimum count of radar ...
1 vote
202 views

### Why does the schema theorem of genetic algorithms hold?

I have been reading about the Schema Theorem - one of the first theorems from the field of evolutionary computing and genetic algorithms, largely responsible for justifying the use of genetic ...
98 views

### Does reaching the global optima guarantee good performance in a task?

It is to my understanding that, in deep learning, we are essentially trying to minimize the loss function that we have defined and reach its global optima through some form of optimization technique. ...
82 views

### Is it possible to find a good neural network structure without training it? [duplicate]

Neural networks consist of so many parameters. Researchers could create as many possible neural networks as they wish. So I want to ask a general question. Could we devise an evolutionary algorithm ...
1 vote