New answers tagged

0

Since it is a trained network already, when you run an example through it, the gradient will not have a very high variance. The gradient varies a lot when you are training a network from the scratch but then it stops varying much since it understands the pattern.


1

I never used a k-WL in practice, but I did apply weisfeiler-lehman for my graph tasks. As you can know, the WL provides the coloring by interactive procedure that's assign each node a 'color' (basically some kind of label reflecting the node neighborhood). Counting colors allows to compare two graphs on isomorphism, but it's not that important here, the key ...


1

We want a distribution over $w$, don't we? Yes. You want to obtain a distribution over the parameters, which models the uncertainty about the parameters. This distribution over the parameters can induce a probability distribution over the possible functions consistent with your data. Why is $a$ integrated out here and not $w$? This is just the definition ...


0

You have two dependent variables $a$ and $w$. So, there is a joint distribution $p(w, a)$. You can make a marginalization by one of them, pretty much as you did in your second formula. $$p(w) = \int p(w, a)da$$ $$p(w) = \int p(w | a)p(a)da$$ The only difference in this case, the calculation made for the specific point $x_i, y_i$, which is empathized by sub-...


0

After working on it for a while this is what I got. Concerning proposition 1 in the paper, a rigorous statement could be the following version of the Gradient Theorem for line integrals: Proposition 1. (Gradient Theorem for Lipschitz Continuous Functions). Let $U$ be an open subset of $\mathbb{R}^n$. If $F : U \to \mathbb{R}$ is Lipschitz continuous, and $\...


1

What this paper is not saying is that the discriminator, $D_{\phi}$, always returns a scalar value of zero. What they are saying is that the generator, $G_{\theta}(z)$, has accurately learned the distribution of the input data, and that the discriminator produces the correct answer for each input from the generator. It's a mathematical description of the ...


1

The lower bound in MINE is as follows: $$\widehat{I(X;Z)}_n = \sup_{\theta\in\Theta} \mathbb{E}_{\mathbb{P}_{XZ}^{(n)}}[T_\theta] - \log{\mathbb{E}_{\mathbb{P}_X^{(n)} \otimes \hat{\mathbb{P}}_Z^{(n)}}[e^{T_\theta}]}$$ Here $\mathbb{\hat{P}^{(n)}}$ denotes the empirical distribution that we get from n i.i.d samples of $\mathbb{P}.$ Note that in the above ...


0

Multiple attention heads in a single layer in a transformer is analogous to multiple kernels in a single layer in a CNN: they have the same architecture, and operate on the same feature-space, but since they are separate 'copies' with different sets of weights, they are hence 'free' to learn different functions. In a CNN this may correspond to different ...


1

This is from my own experience with (Vanilla) GANs, so it might not translate exactly to your application, but maybe it gives some orientation. your learning rate seems quite high. I've quite frequently found that 1e-5 is a good value for me. The training might take longer but will probably be more stable. have you tried using dropout? It's a good ...


Top 50 recent answers are included