According to this paper from FAIR : https://arxiv.org/abs/2110.09348 , contrastive learning methods suffer from the problem of dimensional collapse where "the embedding vectors end up spanning a lower-dimensional subspace instead of the entire available embedding". According to the authors, this problem is caused by two main reason which are strong augmentations and the implicit regularization of deep neural networks that tend to converge to low rank solutions. I am not sure how Openai's CLIP avoids this problem. Is it just the sheer scale and the fact they used a batch size of 32k smaples?

  • $\begingroup$ How do you know for certain that they do avoid it? Maybe its simply also happening in CLIP but because the performance is good, its not problematic? $\endgroup$ Sep 6, 2023 at 17:15


You must log in to answer this question.

Browse other questions tagged .