Constructing a dataset that scores well only for a specific set of hyper parameter values

Question

When designing a machine-learning system, there are various parameters that have to be determined. I am interested in the following general question: is it possible to construct a dataset on which the system will have good performance with some specific set of parameters, but not with other paramteres?

To be more concrete, let's focus on neural networks. Suppose we have a simple neural network: a multilayer perceptron with a single hidden layer. The size of the input is fixed, the activation function is fixed (e.g. tanh), and the output is binary. The only parameter that has to be determined is the size of the hidden layer.

My question is: given a number $n$, is it possible to construct a dataset $D_n$ such that:

• The MLP with $n$ hidden nodes has good performance on $D_n$ (e.g. in 10-fold cross validation);
• The MLP with $n-1$ hidden nodes has bad performance on $D_n$

?

_{Note: I asked in CS theory but got no reply.}

Answer 1

I'm not familiar with any existing, robust methods to generate such a dataset. Here are some thoughts though.

You propose using an MLP with a single hidden layer. That means we have two weight matrices, and two activation functions (one for hidden layer, one for output layer). Some notation:

• $d$: dimensionality of input vectors
• $n$: dimensionality of hidden layer
• You mentioned binary output, so I'll assume that dimensionality is $2$
• $W_n^{(1)} \in \mathbb{R}^{d \times n}$: the first weight matrix in the scenario where we're using $n$ hidden nodes
• $W_n^{(2)} \in \mathbb{R}^{n \times 2}$: the second weight matrix in the scenario where we're using $n$ hidden nodes
• $g$: first activation function
• $h$: second activation function

Then, given an input vector $x \in \mathbb{R}^d$, our neural network will generate output $f_n(x) = h(W_n^{(2)} (g(W_n^{(1)}x)))$.

Now, in general we of course expect Neural Networks to only find a local optimum, but if you want a robust solution you'll want it to be able to handle the worst case, and the worst case for your "adversarial" task is when the Neural Network manages to find the global optimum. So, we'll assume it can find the global optimum.

Essentially, what you're looking for is a dataset $D_n$ containing a number of input vectors $x$, such that:

1. $f_n(x) = h(W_n^{(2)} (g(W_n^{(1)}x)))$ provides good / optimal results (after training to the global optimum)
2. $f_{n - 1}(x) = h(W_{n - 1}^{(2)} (g(W_{n - 1}^{(1)}x)))$ provides poor results (even after training to the global optimum of this setup).

In other words, you want to find a collection of vectors $x$ such that it becomes impossible to find a collection of weights in the $n - 1$ case where $f_{n-1}(x)$ is a good approximation of $f_n(x)$. You want to make it impossible that $f_{n - 1}(x) \approx f_n(x)$.

---

Now this has turned into a clear mathematical problem. I'm not familiar with any established methods in mathematics to solve a problem like this, maybe there are though. My best guess at this point in time would be a procedure like the following:

1. Generate random "ground truth" versions of the weight matrices $W^{(1)*}_{n}$ and $W^{(2)*}_{n}$ (just completely random matrices).
2. Generate completely random input vectors $x$. Compute the corresponding ground truth labels as $f^*_n(x) = h(W_n^{(2)*} (g(W_n^{(1)*}x)))$.
3. Hope that the neural network with $n$ hidden nodes can recover the ground truth weight matrices that were previously generated randomly.
4. Hope that the neural network with $n - 1$ hidden nodes cannot find an accurate approximation.

In theory, the MLP with $n$ hidden nodes should be able to learn the exact ground truth function. In theory, under certain conditions, the MLP with $n$ hidden nodes should not be able to learn the exact ground truth function. I suspect those "certain conditions" would be that the rows/columns of the weight matrices should be linearly independent, which is likely with randomly generated matrices, but I'm not 100% sure on this. Even if it can't learn the exact ground truth, it may still be capable of learning an approximation... there may be ways to find upper bounds on how close such an approximation could get, but I'm not sure.

Answer 2

Hidden layer number is an indicator of the dimension of the weight matrix , so once you train a network with certain number of hidden layer neurons , the weights get fixed at the point you stop training. so changing the number hidden layer neurons makes the previous weights incompatible with it due to change in weight-matrix dimenstion . even if you alter the weight matrix to work with the new network it is again equivalent to a network with randomly assigned weights. so a fully trained networks becomes a new network once you change its hidden layer dimension and gives bad performance. This hase a very little to do with the dataset.