2
$\begingroup$

I am looking for the standard notation to define element-wise / Hadamard-style functions, if there is one.

That is to say, if the operator I am looking for were represented by a hexagon ⬡, I could use it as such:

$$A(x) = \underset{i}{\Large{⬡} } f(x_i)$$

$$A : \mathbb{R}^n \rightarrow \mathbb{R}^n$$ $$f : \mathbb{R} \rightarrow \mathbb{R}$$

It is very convenient to define such functions explicitly because I want to manipulate them: $B \circ A$ . It seems to me that the following notation is correct: $A_i(x) = f(x_i)$ but I worry it is nonstandard and confusing.

My functions are non-linear so I cannot simply apply them directly to the array as a vector.

Related:

$\endgroup$
0
0
$\begingroup$

The mathematical notation for complex tensorial expressions always tries to balance complexity and precision. More precise notation - the one that explicitly spells all the indices - becomes extremely convoluted very quickly. My favorite example illustrating it is from physics -- the Standard Model Lagrangian is written shortly on T-shirts and coffee mugs as:

$$ \mathcal{L} = - \frac{1}{4} F_{\mu \nu} F^{\mu \nu} + (i \bar{\psi} \hat{D} \psi + \bar{\psi}_i y_{ij} \psi_j \phi + h.c.) + |D_\mu \phi|^2 - V(\phi) $$

But if you try to expand all the indices in all the objects above - then it barely fits on a page.

On the other hand, more succinct notation always leads to ambiguities in interpretations. Your example $f(x_i)$ can be read as: $$f(x_0, x_1, \dots, x_N)\quad \text{or as}\quad f(x_0),f(x_1), \dots, f(x_N)$$
One way to implicitly resolve this ambiguity is to show that the index $i$ "escapes" the argument brackets:

$$a_i = f(x_i)\quad \text{or e.g.} \quad \sum_if(x_i)$$

This can only be interpreted as $f(x_i)$ being element-wise. Also, at least in my opinion, using $x_i$ with index and $x$ without index in the same expression is extremely confusing.

And, of course, the best way to resolve these ambiguities is to state them explicitly. For example, I've seen authors using square brackets $f[x_i]$ or capital letters $F(x_i)$ the vector-argument functions.

$\endgroup$
2
  • $\begingroup$ Hi Kostya thank you for your answer. Sorry, I think my question may not be clear, f(x_i) is not a vector function, it is a simple scalar function. f: R->R ; a: R^n->R^n . Unless I miss something, I have to have a variable index and unindexed in the same expression, because I am relating behaviour an the variable to its components? If I just write a_i = f(x_i) it looks like x is a free variable. $\endgroup$
    – user7834
    May 19 at 19:25
  • $\begingroup$ I have edited my question to follow your advice regarding capital letters. $\endgroup$
    – user7834
    May 19 at 19:31
0
$\begingroup$

Based on my experience, I would say that the standard notation is just to have a regular function, and specify that it applies element wise. For example, a common notation for activation functions is $\sigma$, so e.g. you could represent the activations of a regular dense layer as $\sigma(W x + b)$ where $x, b$ are vectors and $W$ is a matrix. I've never seen a special notation for specifying that the function $\sigma$ is applied element wise.

As you suggest in the question, if the function to be applied element wise is linear, then you can use either hadamard product, e.g. $a \circ x$ or the diag function $\text{diag}(a)x$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.