1
$\begingroup$

I would really appreciate if someone could comment the following method of training neural nets providing them with some meta data (Making them more color prone only if needed, whereas now they're mostly silhouette / outline aware). (Comment and especially giving a reference to some papers, preventing me from reinventing the wheel.)

But let's start at the very begining and say that we're performing simple image recognition and each image has 24 bit color depth, simply 1 Byte per each RGB channel. I'm more eager to use usually bigger pics, sacrifing color quality, however not in all cases (that statement is crucial in this question).

To limit the computational burden I'm NOT keen on using the full information about color (3 Bytes per pixel), but rather shrink it to 1 Byte per pixel and here is the catch:

I'm reluctant neither to use gray scale nor to cast original tints to single (common among all pics) color palette of 256 hues. So I came up with an idea of reversing the method called debayering or demosaicing image from Color Filter Array data).

To achive it, for every pixel only one color channel is preserved. Because of the human perception of colors, green component is overrepresented covering 50% of pixels, so 25% left for blue and red. In this particular example below, the upper left pixel correspond to Blue, followed by Green, then Blue, next Green and so on to the end of the row. The second uppermost row starts with Green, afterwards Red, one more Green, Red and also repeating till the end of line. This horizontal patterns are altering with the parity of the row number, which is nicely depiced on https://en.wikipedia.org/wiki/Bayer_filter from where I've got following graphics:

enter image description here

To better illustrate this method I'm using the thumbnail of famous Mona Lisa painting with its grayscale version next to it. (By the way, it isn't in my training set, but is familiar to everyone).

OriginalOriginal B&W

A greenish leftmost image is the result of applying the reverse debayering / demosaicing CFA method. This picture consist of pixels that are either Blue or Red or Green with different brightness level. In the browser window that could be poorly visible, however if you download this image and magnify it substantially, the patter would be revealed.

Let's say that in the original picture one can find a small square of 2x2 pixels, all of them representing a light skintone 0xF4D374 (in hex). In this grouping, 2 green pixels would be chopped into green channel and will get a value of 0xD3, the blue-related will get a value of 0x74 and the remaining red would get 0xF4. In the leftmost image below the corresponding pixels were presented by hex colors: 0x00D300, 0x000074 and 0xF40000 respectively, whereas in the right picture exactly the same values (0xD3, 0x74, 0xF4) were shown in grayscal (of 256 possible shades).

CFA appliedCFA applied plus B&W

After this color-flattening, our input batch has shrinked by two-thirds and at the same time original colors can be more-or-less restored (of course not lossless, but well enough).

However, I don't suppose that anyone had a problem with recognising this picture after transformation. Likewise, all my models could be well trained to recognize outline/silhouette of the object, but they require a way lot more training data (at least one-two orders of magnitude) to be color-aware.

The ultimate question is, how to design models that would threat shape and colors in similar manner. Maybe that would be not in 100% mathematically proper, but shape and color must be orthogonal.

Nevertheless, I don't want to always decode the color, but only if its needed - in ealier epoch it learned sillhouettes/shapes and that there're many similar objects in this regard, so in the next epoch it should pay also / more attention to tints.

Have you encountered articles about such method of using color if object demarcation / labelling process cannot be based only on shape? I would be really gratefull for any paper or other reference.

I'm rather newbie to neural nets, so sorry if this is something widely known to everyone but me ;-)

Thanks in advance for any hint.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.