0
$\begingroup$

I don't get $G^(i)$the set of indices into the inputs for group $i$, $\{(i −1)k+ 1, . . . , ik\}$ when creating a maxout units/function, these thing that outputs the maximum element of groups: $$g(z)_i= max_{[j∈G^{(i)}]}z_j$$

Maxout units(Goodfellow et al., 2013a) generalize rectified linear units. Instead of applying an element-wise function $g(z)$, maxout units divide $z$ into groups of $k$ values. Each maxout unit then outputs the maximum element of one of these groups:

$$g(z)_i= max_{[j∈G^{(i)}]}z_j$$

where $G^{(i)}$is the set of indices into the inputs for group $i$,$\{(i −1)k+ 1, . . . , ik\}$.

This provides a way of learning a piecewise linear function that responds to multiple directions in the input $x$ space. A maxout unit can learn a piecewise linear, convex function with up to $k$ pieces. Maxout units can thus be seen as learning the activation function itself rather than just the relationship between units. With large enough $k$, a maxout unit can learn to approximate any convex function with arbitrary fidelity. In particular, a maxout layer with two pieces can learn to implement the same function of the input $x$ as a traditional layer using the rectified linear activation function, the absolute value rectification function, or the leaky or parametric ReLU, or it can learn to implement a totally different function altogether. The maxout layer will, of course, be parametrized differently from any of these other layer types, so the learning dynamics will be different even in the cases where maxout learns to implement the same function of $x$ as one of the other layer types.

Deep Learning, Bengio and al. page 190

I tried to work it out with a concrete example:

Let's take a concrete example with k = 3. In this case, the groups would be as follows:

$$G^{(1)} = {(1 - 1)3 + 1, ..., 1 * 3} = \{1, 2, 3\}$$ $$G^{(2)} = {(2 - 1)3 + 1, ..., 2 * 3} = \{4, 5, 6\}$$ $$G^{(3)} = {(3 - 1)3 + 1, ..., 3 * 3} = \{7, 8, 9\}$$ And so on...

So, in this example, if we have 9 inputs, we'd have 3 maxout units, and each maxout unit would use 3 inputs to calculate its output, choosing the maximum from these 3 values.

Suppose we want to model the cost of producing a certain number of units of military weapons as a function of the quantity produced. We can use a piecewise linear function for this.

So let's imagine that the production cost is defined by a convex function with three pieces (k = 3):

  • For the first 100 units produced, the cost is \$10,000 per unit.
  • For 101 to 200 units, the cost is \$8,000 per unit.
  • For 201 units and above, the cost is \$7,000 per unit.

So here, I guess each maxout unit would take as input the quantity produced, and each group $G^(i)$ would be associated with one of the pieces of the function.

For example, if we wanted to calculate the cost of production for 150 units, I'm not sure but I guess we'd have :

$G^{(1)} = \{1, 2, 3\}$ corresponding to the first part of the function (\$10,000 per unit). $G^{(2)} = \{4, 5, 6\}$ corresponding to the second part of the function (\$8,000 per unit).

I don't get what are $\{1, 2, 3\}$ and $\{4, 5, 6\}$ standing for, concretely

If I apply the maxout function for each group, I guess it means that maxout unit 1 will select the maximum of the values associated with $G^{(1)}$, and maxout unit 2 will select the maximum of the values associated with $G^{(2)}$, and so on ... but then it doesn't make any sens

Indeed, for 150 units, I guess maxout 1 would give a value of \$10,000 per unit, and maxout 2 would give a value of \$10,000 per unit, ... I don't know how I am expected to calculate the total cost of production then ...

I am a slow but passionate learner in math and deep learning. Don't hesitate to explain it to me as if I was a teenager.

$\endgroup$

0

You must log in to answer this question.