How do transformer-based architectures like Roberta generate contextual embeddings? The articles I've read keep saying that transformer encoders work bidirectionally. Because of self-attention, they can look at every token, unlike RNN/LSTM, which can only process the previous hidden state. I'm not sure how the Transformer accomplishes that.



You must log in to answer this question.