How to perform insect classification given two images of the same insect?

Question

I'm relatively new to image classification. Currently, I am trying to classify insect images, using a convolutional neural network (CNN). When I ask a human expert to identify an insect, I usually provide 2 photos: back and face. It seems that sometimes one feature stands out and allows identification with high certainty ("spots on the back - definitely a ladybug"), while other times you need to cross-reference both angles ("grey back could mean a few things, but after cross-referencing with the eyes - it's a moth").

How is it customary to implement this? Naively I was considering:

1. Two separate networks, one for backs and one for faces? If so, what formula is best for weighing in their outputs?

2. Single network, but separate dual classifications - e.g. "moth face", "moth back", "ladybug face", "ladybug back"?

3. A single network, feed everything naively (e.g. moths from different angles, all with the same classification "moth") and rely on the NN to sort it out itself?

Answer 1

There are several ways you can do this.

One is to input both images in input, so it can be a 2 input system or an input with 6 channels.

As you suggested in 1st point, you can make 2 networks, connect them at the end and add another layer for final classification or use outputs from both and train another classifier (like Gradient bosting). You can look up ensemble techniques and ensemble of neural networks for more.

For dual classification and naive approach, you will again have a problem in case there are some insects that can solely be identified from features at a specific angle, if so you can then discard other angles (e.g. ignore moth face if moth back is the distinguishing feature)