All Questions
1 question with no upvoted or accepted answers
0
votes
0
answers
26
views
Any LMs that use tanh (generalization) instead of sigmoid within Attention?
Question is in the title. Posts such as this and this mention how this would be possible. I have some colleagues who have anecdotally heard of this being done on very small transformer models but I ...
- 11