I have my own data to train a logistic regression model (for a multi-class classification task), and I want to know how the distribution of weight parameters changes after each update with gradient descent.

For example, let's say that there are $f$ many features for each input, and the weight $W$, which is a $c \times f$ matrix, where $c$ is a number of classes, is initialized with uniform distribution $U(-1/\sqrt{f}, 1/\sqrt{f})$, which is LeCun uniform initialization.

For each step of gradient descent with Cross-Entropy loss, it will be updated as $$ W_{t+1} = W_{t} - \alpha \frac{\partial \mathcal{L}}{\partial W_{t}} $$ where $\alpha$ is a learning rate and the gradient is given $$ \frac{\partial \mathcal{L}}{\partial W_{t}} = \frac{1}{n} (\mathbf{y} - \mathbf{p})^{T}\mathbf{X} $$ where $\mathbf{X} \in \mathbb{R}^{n\times f}$ is an input matrix, $\mathbf{y} \in \mathbb{R}^{n \times c}$ is one-hot encoded labels, and $$\mathbf{p} = \mathrm{softmax}(\mathbf{X}W_{t}^{T}) \in \mathbb{R}^{n \times c}$$ is the model's output (predicted probability for each example & class).

What I want to know is how the distribution of $W_{t}$ changes as $t$ increases, if some information about $\mathbf{X}$ is known. More precisely,

  1. Is it possible to get a bound of $\mathbb{E}[||W_{t}||_{\infty}]$ in terms of $t, \alpha, ...$? We may assume that the input $\mathbf{X}$ is also bounded (in the sense that $||\mathbf{X}||_{\infty} \leq M$ for known $M$.) I have a really rough bound for $||W_{t}||_{\infty}$, but it is not good enough.
  2. Are there any works in this direction for other models, such as MLPs?

When I plotted the value $||W_{t}||_{\infty}$ for each $t$, then it seems that it increases sub-linearly in $t$, but


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.