New answers tagged pytorch
0
votes
What is the Intermediate (dense) layer in between attention-output and encoder-output dense layers within transformer block in PyTorch implementation?
So I came across this myself when trying to explore why models require intermediate layers in general. BERT is fundamentally different than the previous transformer architectures. The intermediate ...
2
votes
Accepted
Is there any advantage to providing multi-dimensional input to torch modules?
Yes, for some applications, the spatial component of the tensor is indeed very important, but not for the examples you mentioned. The first point to clarify is that even though the ...
- 453
0
votes
How to properly optimize shared network between actor and critic?
A common practice involves using a shared encoder, which is updated based solely on critic loss, as implemented in DrQv2.
- 51
0
votes
While fine-tuning a decoder only LLM like LLaMA on chat dataset, what kind of padding should one use?
Decoder only text LLMs are autoregressive. Like any autoregressive function if you do not left pad there will be no prior token to your intial token for efficient gradient descent.
In most text ...
Top 50 recent answers are included