Skip to main content
Share Your Experience: Take the 2024 Developer Survey

New answers tagged

1 vote

What is $z|y$ in Conditional Adversarial Nets?

Yes, they might be abusing a bit the notation, however the meaning is close to what that notation would like to say. $x|y$ and $z|y$ just mean that $x,z$ might be the same for multiple samples, only ...
Alberto's user avatar
  • 2,173
1 vote

Can anyone please explain the Recurrent Neural Network calculation shown in the picture?

This is the RNN architecture. Image is taken from https://en.wikipedia.org/wiki/Recurrent_neural_network. Here $U = B$, $V = A$, and $W = C$. We are using ReLU as the activation function. $\phi(x) = ...
vishal's user avatar
  • 11
1 vote

Determining optimal data size for generalization in transformer encoders, particularly for Time-Series signal data

In order for a neural network to generalize well, it has to be trained on tons and tons of data, otherwise if the model is shown very few samples of the training data, it will be biased towards these ...
Leo's user avatar
  • 425
10 votes
Accepted

When to use Tanh?

Using tanh in hidden layers require careful initialization of network weights and works best with the input features normalized within the same range as output (i.e. -1 to 1). It have expensive ...
Leo's user avatar
  • 425
1 vote

Last linear layer of the decoder of a transformer

Well well, this text in the link posted in one of the answers above (https://www.tensorflow.org/text/tutorials/transformer) gives the answer to the question. Note: The model is optimized for ...
eddys's user avatar
  • 111
0 votes

How does high entropy targets relate to less variance of the gradient between training cases?

Suppose you have two tokens that are equally likely. In your original training data, your next-token predictions are $$[1, 0]\quad\text{and}\quad [0, 1]$$ an equal number of times. The entropy of each ...
programjames's user avatar

Top 50 recent answers are included