1
$\begingroup$

I need to implement a rule and have defined a lower triangular boolean mask for the weights that I want to keep static for a zero value. In which condition triangular weight matrix will be used?

$\endgroup$
1
  • $\begingroup$ Triangular weight matrix could be much weaker to approximate complicated functions, unless your computing resources are limited you may try it. $\endgroup$
    – cinch
    Dec 20, 2022 at 17:54

2 Answers 2

1
$\begingroup$

I did some testing on nanoGPT (https://github.com/karpathy/nanoGPT) using triangular weights matrices instead of full one and the result were interesting : using triangular matrices seems to have a very limited impact on the learning (for gpt2). enter image description here

(black and orange curve are the learning for triangular matrix and the pink one is the full linear one)

$\endgroup$
2
  • 1
    $\begingroup$ I would bet It perform the same as dropout 0.5 $\endgroup$ Mar 30 at 19:31
  • $\begingroup$ My bet (but i'm not sure) that it will not be the case ... dropout randomly choose some activation to shut down. With triangular matrix you have a deterministic behaviour. Also the advantage of triangular is that you can hope to find better gpu kernel to do your triangular matrix multiplication (but I will not do that, I will not code in cuda :)). $\endgroup$ Apr 1 at 9:39
0
$\begingroup$

Constraining weights in a neural network, either a hard constraint like your idea, or a soft constraint that adds to the cost function, is often a form of regularisation. By limiting the form of acceptable solutions, regularisation makes it harder, often impossible, to exactly match training data - this is a good thing because what you most often want to learn is a generalisation of the examples that works well with new data. Forcing a machine learning process into a space where it can approach but never reach a perfect solution can make good generalisation.

That means you might use such an idea to combat overfitting. The utility of the idea will vary depending on the problem. There may be no general use for it, because it's never better than other well established approaches. But there could be some problems it is particularly suitable for. You would have to try it and see.

The main problem I see with the approach in general is that it's all or nothing. You cannot easily have an arbitrary fraction of the diagonal mask.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .