3

I wonder how expert AI researchers deal with that, do they perform multiple experiments, even if this takes extremely long? Do they draw conclusions from single runs? Unfortunately, the question you ask in the main body of your question here ("how do expert AI researchers do things") often turns out to actually be different from the question in ...


3

I suggest you take a look at Chris Olah's blog. Has several interesting post including ones on visualizing weights and interpretability. Most of his papers also have Google Colab links so you can reproduce the results. If you want something more similar to the model.summary() method you mention, TensorBoard Graph Dashboard might help.


2

This width of a neural network *layer is an agreed upon term. *The width of a neural network is generally the width of the widest layer of the neural network. *I would caution how you use the phrase "width of a neural network" due to interpretability and scale, *and the fact that neural networks often contain layers with varying numbers of neurons, ...


2

Gradient clipping is a technique that tackles exploding gradients. The idea of gradient clipping is very simple: If the gradient gets too large, we rescale it to keep it small. More precisely, $$ \text{if } \Vert \mathbf{g} \Vert \geq c, \text{then } \mathbf{g} \leftarrow c \frac{\mathbf{g}}{\Vert \mathbf{g} \Vert} $$ where $c$ is a hyperparameter, $\mathbf{...


1

Technically, nothing prevents you from doing so. When you have mulitple losses, you may call .backward() at each term separately. However, I wonder, whether it makes sense to optimize each individual path as a separate objective, since if we have multiple of them - we would like to solve several tasks simultaneously. Probably, it could be beneficial as some ...


1

I am not sure if the process defined in the question is meaningful at all. If you mean to simply add the contribution of each $L$ without running the algorithm for the mini-batch, it makes no difference at all if you make one update or more; as the loss functions' contribution are simply added in the update. If on the other hand you mean to run the algorithm ...


1

I would distinguish at least 2 cases when it comes to a generic expression like prior knowledge: generic extra information provide to a model, really close if not the same as feature engineering. literal prior probability distributions used to initialize or guide a model during training. For the first case there's plenty of examples that we can provide. ...


1

In the usual scenario, case 2 occurs. In the deep learning frameworks, Tensors have special dimension (usually corresponding to the 0 axis) which numerates the example in the batch. Look for example in the PyTorch documentation of Conv2d or Tensorflow documentation of Conv2d. The same is true for any Layer - Linear, MultiheadAttention, RNN. All samples from ...


1

I think there's a crucial point missed in the question, touched by jros answer but without further elaboration. If you train a model on domain A: single lightning condition and test it on domain B: two lightning condition then you're not evaluating generalization but transfer learning capabilities. Or to phrase it differently you're evaluating how close ...


1

In addition to the other good answer, if you can afford the luxury of running your experiments multiple times, you can also use hypothesis testing to test whether there is any significant difference between the performance (e.g. accuracy) of the two models. Hypothesis testing is not widely used (or, at least, reported in research papers) in the ML/DL ...


1

Tricky question. In my experience is better to just look for math resources on classic upsampling method, since deep learning papers and books tend to give them for granted, or not something related to AI (they are after all analytic methods). Another reason is probably that the math is not that hard, and already the wikipedia pages offer a good description ...


1

Simply, it is just a design choice. Isotropic gaussian is one of the easiest density to work with. It has an easy-to-compute likelihood and easily reparameterizable. You are free to use other distribution, but might face computational or implementation hurdles.


1

This is just an implementation issue. One reason is the Huggingface implementation (which is not the original implementation by Google) wants to strictly separate the tokenization from the modeling. It is a convention that the input sequences are zero-padded, but in theory, it does not have to be so. In the Huggingface implementation, you use a different ...


Only top voted, non community-wiki answers of a minimum length are eligible