Is color information only extracted in the first input layer of a convolutional neural network?

In a convolutional neural network (CNN), since the RGB values get multiplied in the first convolutional layer, does this mean that color is essentially only extracted in the very first layer?

A snippet from CS231n Convolutional Neural Networks for Visual Recognition:

One dangerous pitfall that can be easily noticed with this visualization is that some activation maps may be all zero for many different inputs, which can indicate dead filters, and can be a symptom of high learning rates.

Another one.

Typical-looking activations on the first CONV layer (left), and the 5th CONV layer (right) of a trained AlexNet looking at a picture of a cat. Every box shows an activation map corresponding to some filter. Notice that the activations are sparse (most values are zero, in this visualization shown in black) and mostly local.