Commonmark migration
Source Link

I think they use the Xavier/Glorot's initialization method. You can read from [the original paper][1]the original paper:

We initialized the biases to be 0 and the weights $W_{ij}$ at each layer with the following commonly used heuristic:

$W_{ij} \sim U [ -\frac{1}{\sqrt{n}}, \frac{1}{\sqrt{n}}] $

where $U[−a, a]$ is the uniform distribution in the interval $(−a, a)$ and $n$ is the size of the previous layer (the number of columns of $W$)

Some people use this as some reports said this initialization method lead to better result [1]: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf

I think they use the Xavier/Glorot's initialization method. You can read from [the original paper][1]:

We initialized the biases to be 0 and the weights $W_{ij}$ at each layer with the following commonly used heuristic:

$W_{ij} \sim U [ -\frac{1}{\sqrt{n}}, \frac{1}{\sqrt{n}}] $

where $U[−a, a]$ is the uniform distribution in the interval $(−a, a)$ and $n$ is the size of the previous layer (the number of columns of $W$)

Some people use this as some reports said this initialization method lead to better result [1]: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf

I think they use the Xavier/Glorot's initialization method. You can read from the original paper:

We initialized the biases to be 0 and the weights $W_{ij}$ at each layer with the following commonly used heuristic:

$W_{ij} \sim U [ -\frac{1}{\sqrt{n}}, \frac{1}{\sqrt{n}}] $

where $U[−a, a]$ is the uniform distribution in the interval $(−a, a)$ and $n$ is the size of the previous layer (the number of columns of $W$)

Some people use this as some reports said this initialization method lead to better result

Source Link
malioboro
  • 2.4k
  • 2
  • 11
  • 36

I think they use the Xavier/Glorot's initialization method. You can read from [the original paper][1]:

We initialized the biases to be 0 and the weights $W_{ij}$ at each layer with the following commonly used heuristic:

$W_{ij} \sim U [ -\frac{1}{\sqrt{n}}, \frac{1}{\sqrt{n}}] $

where $U[−a, a]$ is the uniform distribution in the interval $(−a, a)$ and $n$ is the size of the previous layer (the number of columns of $W$)

Some people use this as some reports said this initialization method lead to better result [1]: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf