No, channels do not have to only represent colours. It is common for them to represent other things, even without considering feature maps. For instance RGBD images, where D is a depth measurement or distance from a sensor. Or when CNNs are applied to grid-based games, such as chess or go with AlphaZero, where the input channels are information about game pieces on a board.
Mathematically, there is little to differentiate between a channel or a feature map. Both are numerical values stored in some muti-dimensional array, most often with the following assumptions:
All values in a single channel or single feature map represent a measurment of the same concept. That might be how much blue light was detected at a sensor at a point in space, or it may be the degree to which pixels close to that point match patterns associated with the centre of a certain type of cat's nose.
The values are considered co-located with other feature maps or channels within the same system, such that values at index $i,j$ (or just $i$ of 1D or $i,j,k$ for 3D etc) in one channel are considered to be at same location as values in related channels or feature maps, at least within the same layer.
You will tend to find channels used to describe inputs and outputs that can be directly visualised, whilst feature maps tend to be used to describe the more abstract pattern matching that occurs in the outputs of a CNNs hidden convolutional layers. However, the two terms can be used loosely, and sometimes interchangeably.
The feature maps within a CNN typically do not carry separate colour channels. Although it is possible to design architectures that keep colour information separate, this is very rarely used - normal CNN architectures allow mixing of all layer channels/features with each new layer, through the mechanism of having weights that connect every input channel/feature to every output channel/feature between layers.
You will sometimes see colour channel information extracted from the neural network weights of the first convolutional layer, in order to visualise what that layer is matching to. That is because the first layer's weights (and only the first layer's weights) can be interpretted as template matching to the input channel for each output feature map. This is not the same as visualising the output feature maps - whilst those maps are influenced by the input colour channels, thus do in a general sense carry colour information, they do not measure colour intensity in the same sense as an image channel used for input.
More generally, because human perception is strongly tied to RGB colour channels, and because computer displays and image formats are designed around this, whenever you see any representation of what a CNN layer is doing, you will see one of:
A greyscale representation of feature map values. This is the closest to "true" representation, but sometimes it is not very informative.
A heat map of feature map values. Using colour may help with visualisation, but it is false colour in the sense that the same colours are not actually in the feature map.
A representative input that would cause the feature map to activate. This can be informative about the feature map, but it is not showing what the feature map is doing directly, and the channels defined in the input are used for colour.
