Timeline for Effects of ReLU Activation on Convexity of Loss Functions
Current License: CC BY-SA 4.0
7 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jan 29, 2022 at 18:37 | comment | added | spiridon_the_sun_rotator | one needs to specify for convexity with respect to which variable the function is. I spoke about convexity with respect to the input arguments of the loss function. $\mathrm{MSE} = \frac{1}{2}(y - \hat{y})^2$ is convex with respect to the prediction $y$. But if you are speaking about convexity with respect to the NN weights, the resulting function generally will be non-convex. | |
| Jan 29, 2022 at 16:54 | comment | added | Edoardo Guerriero | Convex in their formulation, but when applied in neural networks they lose this property, for details check this thread math.stackexchange.com/questions/2402455/… | |
| Jan 29, 2022 at 6:21 | comment | added | spiridon_the_sun_rotator | I would disagree, that every loss function is non-convex. Mean squared error and Crossentropy (Binary or Multilabel) are convex functions. | |
| Jan 28, 2022 at 16:23 | history | edited | Edoardo Guerriero | CC BY-SA 4.0 |
added 46 characters in body
|
| Jan 28, 2022 at 16:18 | history | undeleted | Edoardo Guerriero | ||
| Jan 28, 2022 at 16:08 | history | deleted | Edoardo Guerriero | via Vote | |
| Jan 28, 2022 at 16:05 | history | answered | Edoardo Guerriero | CC BY-SA 4.0 |