Hot answers tagged

4

There isn't any explicit relation between the batch size and the gradient accumulation steps, except for the fact that gradient accumulation helps one to fit models with relatively larger batch sizes (typically in single-GPU setups) by cleverly avoiding memory issues. The core idea of gradient accumulation is to perform multiple backward passes using the ...


2

GPT-3 is based on in-context learning. It’s common wisdom one can hope that bigger models will yield better in-context capabilities. And indeed, this holds true, in the case of GPT-3 175B or "GPT-3". Neverthless GPT-3 is more powerful than it's predecessors. In some of the tasks, GPT-3 failed miserably. This might be due to the choice to use an ...


1

There is an interesting discussion the progress achieved in this field by far in the paper of Francois Chollet - https://arxiv.org/abs/1911.01547. At the present time, many architectures are able to outperform the human in particular tasks, because they have a strong priors coded into them and ability to process a huge amount of data. However, when it comes ...


1

Here is something I've noticed about humans: We're bad at projecting the future with all of its 2nd, 3rd ... N order effects, and we're REALLY bad a projecting and quantifying risk. So, I'm not sure that you'll get an answer that is anything more than either trivially true ("Chatbots will be commonplace") or a correct but wasn't justified ("We'...


1

PCA works well where data sample space is linear. If data sample space is not linear or it is manifold data then model without PCA may perform better than model using PCA. In the given image you can see, data is manifold. In this type of data, PCA, which is based on projection technique does not work well. That's why we use manifold learning technique to ...


1

PCA can make models worse, imagine data points scattered along two elongated parallel rectangles. The axis with the greatest variation will be parallel to the rectangles but doesn't provide any benefit in classifying the points.


1

This is actually a highly technical term, which has been kind of misused and overgeneralized in many places. What does 'convergence' mean in a literal sense? It simply means that a sequence of terms indexed by $\mathbb{N}$ ($X_1, X_2, X_3,..$) tends to a certain fixed value say $X$ as $\mathbb{N} \rightarrow \infty$, but may not achieve the fixed value. (...


1

Upgrade This film depicts a very plausible near future when drones oversee our lives (e.g. the police use them to fight crime) and common people possess self-driving cars. This is definitely one of the best science fiction movies I have ever watched in my entire life, and I have watched many, such as 2001, Blade Runner, or The Matrix. In fact, these are the ...


1

Yes, it must be taken seriously. There are two main reasons: There is no sharp argument or no-go theorem against the existence of a singularity. It's unclear how fast the singularity could develop, but many authors given a non zero probability to this event (see this reference, it contains different points of view on the singularity by leading experts). The ...


Only top voted, non community-wiki answers of a minimum length are eligible