New answers tagged

1 vote

Do GANs have constant running time?

The question is a bit ill defined... usually when we want some bound on the running time, we have to say with respect to what For example: sorting is O(nlogn) wrt the size of the input Transformer is ...
Alberto's user avatar
  • 1,308
1 vote

how to determine the number of units for dense layer for transfer learning?

In adition to @Alberto answer, Start by the same number of layer and units you removed from original model. If the problem is the same, this will be probably the best solution. After that, if the ...
Jonathan Roy's user avatar
2 votes

how to determine the number of units for dense layer for transfer learning?

Not only the units but also the number of layers... you can reason over something like "how complex is your task", but usually we resort to grid search over some educated guesses (like 2/3 ...
Alberto's user avatar
  • 1,308
1 vote
Accepted

The training process of a conditional GAN

I assume you mean how to label the image and class inputs since the discriminator can reasonably output either "real" or "fake" labels for either of those inputs, and you generally ...
Neil Slater's user avatar
  • 30.2k
2 votes

Are foundation models something fundamentally new? Is there a proper definition?

A foundation model is any model that is trained on broad data (generally using self-supervision at scale) that can be adapted (e.g., fine-tuned) to a wide range of downstream tasks from "On the ...
bogovicj's user avatar
  • 231
3 votes

Can mini-batches for stochastic gradient be balanced but not representative of the training data?

Though you've already accepted an answer, from your comments it seems you're still somewhat unclear and need an example. Take your example of (binary) classification, usually balanced training data ...
Double Knot's user avatar
3 votes
Accepted

Can mini-batches for stochastic gradient be balanced but not representative of the training data?

The problem is, what mathematical definition do you give to "balanced" and "representative"? Balanced usually means that classes are uniformly distributed, but then you are ...
Alberto's user avatar
  • 1,308
0 votes

My model is only improving when learning rate is 1. Should I be worried?

Learning rate must be below 1 because if it were above 1 the output would only depend on the last input of the last training epoch , which cant be correct.
Root's user avatar
  • 1
0 votes

My model is only improving when learning rate is 1. Should I be worried?

From an optimization POV, there is no problem, indeed, given a loss $L$ for which a good learning is $10^{-4}$, you can build a second loss equivalent which has $1$ as good learning rate, just by ...
Alberto's user avatar
  • 1,308
1 vote

My model is only improving when learning rate is 1. Should I be worried?

A learning rate of 1 is indeed atypical in many machine learning contexts, as it may cause the model to converge too quickly and overshoot the global minimum. Generally, a smaller learning rate within ...
Ari Setiawan's user avatar
1 vote

What causes my loss curve to consistently oscillate when training an LLM?

Based on the colab notebook that you used, the periodic oscillation of the training loss does not seem to originate from the learning rate scheduler which is set to "constant". However, I ...
JMarc's user avatar
  • 111
2 votes

the best choice to reduce a features vector

Feature selection -- the case in which the features are highly correlated is the prototypical case in which you want to select a subset of independent features that allows for an equal performance. ...
Peblo's user avatar
  • 31
1 vote
Accepted

What is the potential issue of nested neural networks

Isn't this just a very short recurrent neural network? Same issues apply, although they are less severe since you aren't applying as many recurrent iterations. Once you start "nesting" them ...
NikoNyrh's user avatar
  • 737
1 vote
Accepted

In the conditional GAN (cGAN) architecture, why does the discriminator need conditional variable?

Because otherwise there is no conditioning... consider the case where you condition the generator but not the discriminator: given an image and a label, the generator proposes an image, which will be ...
Alberto's user avatar
  • 1,308
3 votes
Accepted

Regression loss conditioned by the ground-truth values

Your suggestion should work to focus the ML more on larger angle examples. You may want to try a slightly simpler approach of weighting the loss (or the resulting gradient) by a factor depending on ...
Neil Slater's user avatar
  • 30.2k
0 votes

Why does Stable Diffusion use VAE instead of AE?

Stabilizing diffusion uses variational autoencoders (VAEs) instead of autoencoders (AEs) because VAEs allow for the generation of continuously distributed representations in the latent space, which ...
kathy's user avatar
  • 1
0 votes

Why does Stable Diffusion use VAE instead of AE?

To my knowledge, when it comes to stable diffusion, the VQ-VAE is the commonly used method. This differs slightly from vanilla VAE which assumes the encoded features to be a normal distribution and ...
Jack123's user avatar
1 vote
Accepted

Inquiry on Combining Two Neural Networks for unsupervised training: Has This Been Researched?

Based on my knowledge, there has not been an architecture that covers all the points you have described. Significant development has occurred in creating networks that utilize subnetworks with ...
Cesar Ruiz's user avatar

Top 50 recent answers are included