New answers tagged

0 votes

What is the Intermediate (dense) layer in between attention-output and encoder-output dense layers within transformer block in PyTorch implementation?

So I came across this myself when trying to explore why models require intermediate layers in general. BERT is fundamentally different than the previous transformer architectures. The intermediate ...
selachianhandler's user avatar
2 votes

Is there any advantage to providing multi-dimensional input to torch modules?

Yes, for some applications, the spatial component of the tensor is indeed very important, but not for the examples you mentioned. The first point to clarify is that even though the ...
Cesar Ruiz's user avatar
0 votes

How to properly optimize shared network between actor and critic?

A common practice involves using a shared encoder, which is updated based solely on critic loss, as implemented in DrQv2.
XiaoBanni's user avatar
0 votes

While fine-tuning a decoder only LLM like LLaMA on chat dataset, what kind of padding should one use?

Decoder only text LLMs are autoregressive. Like any autoregressive function if you do not left pad there will be no prior token to your intial token for efficient gradient descent. In most text ...
Sam Marvasti's user avatar

Top 50 recent answers are included