7
$\begingroup$

Obviously, finding suitable hyper-parameters for a neural network is a complex task and very problem or domain-specific. However, there should be at least some "rules" that hold most times for filter kernel size?!

In most cases, intuition should be to go for small kernel filters for detecting high-frequency features and large kernel filters for low-frequency features, right? For example, 3x3 kernel filters for edge detection, color contrast stuff, ... and maybe rather something like 11x11 for whole object detection, when the objects are >= 11x11 pixels.

Is this "intuition" more or less generally true? How can we decide which kernel filter sizes should be chosen for a specific problem - or even for one specific convolutional layer?

$\endgroup$
2
$\begingroup$

Take a look at this article. It give tools to actually understand what your filters have learn and show what you can do next to optimize your hyper-parameters. Also check more recent articles that seek to provide interpretations of what NN learn.

$\endgroup$
1
$\begingroup$

Welcome to main question of deep learning. There is no valid, single solution which fits to all problems. There are some patterns though, like starting with few filters in early layers and increase filter count while reducing the sizes as you mentioned. That's why it is considered under "research" now. General approach could be to follow is to start with an existing network architecture and check if it provides enough solution for you. However, if you want to modify, you can increase / decrease filter counts and check if the accuracy is getting affected or not.

$\endgroup$
1
$\begingroup$

One key to the answer is in the question, "Even for one specific conv layer." It is not a good idea to build deep convolution networks on the assumption that a single kernel size most aptly applies to all layers. When perusing the configurations that proved successful in publications, it becomes apparent that configurations that varying through their layers are more commonly found to be optimal.

The other key is to understand that two layers of 11x11 kernels have a 21x21 reach, and ten layers of 5x5 kernels have a 41x41 reach. A mapping from one level of abstraction to the next need not be completed in one layer.

Generalities regarding kernel sizes exist, but they are functions of the typical input characteristics, the desired output of the network, the computing resources available, resolution, size of the data set, and whether they are still images or movies.

Regarding input characteristics, consider this case: The images are shot with a large depth of field under poor lighting conditions, such as in security scenarios, so the aperture of the lens is wide open, causing objects at some ranges of distance to be out of focus, or there can be motion blur.

Under such conditions a single 3x3 kernel will not detect many edges. If the edge may span five pixels, the choice exists as to how many layers are dedicated to its detection. What factors affect that choice is based on what other characteristics exist in the input data.

Expect that as acceleration hardware develops (in VLSI chips dedicated to this purpose) that the computing resource constraints will decrease in priority as a factor in kernel size selection. Currently, the computation time is significant and forces the decision about how to balance layer count and layer size to be mostly a matter of cost.

This question begs another question. Can an oversight machine learner learn how to automatically balance the configuration of deep convolution networks? It could then be re-executed whenever additional computing resources are provisioned. It would be surprising if there weren't at least a dozen labs working on exactly this capability.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.