Label arrangement for custom Keras image generator

Question

I am trying to generate 90 and 270 degrees rotated versions of my sample images on the fly during training. I found an example and modifying it. But I am confused about what should be the order? For instance in one batch I have 32 images and my image generator should return total 64 images. Let's say upper case letters are 90 degree and lower case letters are 270 degree rotated images. Should the order be AaBbCc or ABCabc? I apply the same to the validation set. Here is the related code fragment:

Edit: Code fragment added.

    def _get_batches_of_transformed_samples(self, index_array):
    # create list to hold the images
    batch_x = []
    # create list to hold the labels      
    batch_y = []
    # rotation angles
    target_angles = [0, 90, 180, 270]
    angle_categories = list(range(0, len(target_angles)))
    self.classes = target_angles
    self.class_indices = angle_categories
    # generate rotated images and corresponding labels
    for i, j in enumerate(index_array):           
        is_color = int(self.color_mode == 'rgb')
        image = cv2.imread(self.filenames[j], is_color)
        if is_color:
            image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)                               

        for rotation_angle, cat_angle in zip(target_angles, angle_categories):
            rotated_im = rotate(image, rotation_angle, self.target_size[:2])
            if self.preprocess_func: rotated_im = self.preprocess_func(rotated_im)
            # add dimension to account for the channels if the image is greyscale
            if rotated_im.ndim == 2: rotated_im = np.expand_dims(rotated_im, axis=2)                                
            batch_x.append(rotated_im)
            batch_y.append(cat_angle)

    # convert lists to numpy arrays
    batch_x=np.asarray(batch_x)
    batch_y=np.asarray(batch_y)

    batch_y = to_categorical(batch_y, len(target_angles))            
    return batch_x, batch_y

I actually rotate them as 0, 90, 180 and 270 degrees. As seen in the code, for each batch I return all the rotated versions of all the images in the batch. But is this correct or should I return first 0 degree rotated versions, second 90 degree rotated versions so on?

Edit2: I checked my previous work which I use the built in Keras ImageDataGenerator. generator.classes returns [zeros(100,1); ones(100,1)]. In that study I only have two classes. I understand that Keras indexes the images as [class1, class2, ...]. I think I have to do the same.

Answer 1

I am not allowed to comment, so I am adding it here. Not sure I understand, are you asking if the training data must be ordered?

If so, training data must be random, and not AaBbCc or ABCabc. Here the input to the NN will be the source image, and a category 90 degrees or 270 degrees as too separate inputs. i.e. inputs [0...k][k+1][k+2], where 0 - k are normalised pixel data, k+1 is 1 for 90 degree rotation else 0, and k+2 is 1 for 270 degree rotation else 0. Alternatively for more rotation options lets make a = input [k+1] and b = input [k+2]. If a = 0 and b = 0 then 0 degrees, if a = 0 and b = 1 then 90 degrees, if a = 1 and b = 0 then 180 degrees if a = 1 and b = 1 then 270 degrees.

From your image set, draw 3 samples. Sample 1 [training set] must be say 50% of the images, sample 2 [validation set] say 30% of the images and sample 3 [out of sample set] 20% of the images.

Use the training images from sample 1 to update the weights. Read in images from sample 1 at random, the rotation must be random, use your image rotator to rotate the image, and compare it to the output of the neural network, calculate the sum of the mean error and perform gradient descent to update the weights. Repeat the process. After every x weight updates randomly select a few images from the validation set and calculate an error value. If the average error value of the validation images are above some threshold then stop, or if the error begins to get worse. Do not use the validation set to update the weights

At the end use the out-of-sample set of images to verify the performance of the neural network.

The process above does not include hyper-parameter optimisation, or any other optimisation techniques, but does describe the basic way of training a NN using supervised learning.

Edit: I assumed that you wanted to NN to learn to rotate the image. After your updates, I assume you want rotated images to increase generalisation of your NN. In which case randomising or shuffling the input batch is preferred. i.e. send your images to Keras to rotate them, get the batch back any order is fine ABab or AaBb and then shuffle them. Pass the shuffled training and validation set to the NN and compare it to the known label for weight updates.