I need to implement a rule and have defined a lower triangular boolean mask for the weights that I want to keep static for a zero value. In which condition triangular weight matrix will be used?
I did some testing on nanoGPT (https://github.com/karpathy/nanoGPT) using triangular weights matrices instead of full one and the result were interesting : using triangular matrices seems to have a very limited impact on the learning (for gpt2).
(black and orange curve are the learning for triangular matrix and the pink one is the full linear one)
Constraining weights in a neural network, either a hard constraint like your idea, or a soft constraint that adds to the cost function, is often a form of regularisation. By limiting the form of acceptable solutions, regularisation makes it harder, often impossible, to exactly match training data - this is a good thing because what you most often want to learn is a generalisation of the examples that works well with new data. Forcing a machine learning process into a space where it can approach but never reach a perfect solution can make good generalisation.
That means you might use such an idea to combat overfitting. The utility of the idea will vary depending on the problem. There may be no general use for it, because it's never better than other well established approaches. But there could be some problems it is particularly suitable for. You would have to try it and see.
The main problem I see with the approach in general is that it's all or nothing. You cannot easily have an arbitrary fraction of the diagonal mask.