I've solved many problems with neural networks, but rarely work with images. I have about 18 hours into creating a bounding box regression network and it continues to utterly fail. With some loss functions it will claim 80% accuracy during training and validation (with a truly massive loss on both) but testing the predictions reveals a bounding box that only moves one or two pixels in any given direction and seems to totally ignore the data. I've now implemented a form of IoU loss, but find that IoU is pinned at zero... which is obviously true based on the outputs after training. :). I'd like someone to look this over and give me some advice on how to proceed next.
What I Have
I am generating 40000 examples of 200x100x3 images with a single letter randomly placed in each. Simultaneously I am generating the ground truth bounding boxes for each training sample. I have thoroughly validated that this all works and the data is correct.
What I Do To It
I am then transforming the 200x100x3 images down to greyscale to produce a 200x100x1 image. The images are then normalized and the bounding boxes are scaled to fall between 0 and 1. In simplified form, this happens:
x_train_normalized = (x_data - 127.5) / 127.5 y_train_scaled = boxes[:TRAIN]/[WIDTH,HEIGHT,WIDTH,HEIGHT]
I've been through this data carefully, even reconstituting images and bounding boxes from it. This is definitely working.
To train, after trying
mse and many others, all of which fail equally badly, I have implemented a simple custom IOU loss function. It actually returns
-ln(IoU). I made this change based on a paper since the loss was (oddly?) pinned at zero over multiple epochs.
import tensorflow.keras.backend as kb def iou_loss(y_actual,y_pred): b1 = y_actual b2 = y_pred # tf.print(b1) # tf.print(b2) zero = tf.convert_to_tensor(0.0, b1.dtype) b1_ymin, b1_xmin, b1_ymax, b1_xmax = tf.unstack(b1, 4, axis=-1) b2_ymin, b2_xmin, b2_ymax, b2_xmax = tf.unstack(b2, 4, axis=-1) b1_width = tf.maximum(zero, b1_xmax - b1_xmin) b1_height = tf.maximum(zero, b1_ymax - b1_ymin) b2_width = tf.maximum(zero, b2_xmax - b2_xmin) b2_height = tf.maximum(zero, b2_ymax - b2_ymin) b1_area = b1_width * b1_height b2_area = b2_width * b2_height intersect_ymin = tf.maximum(b1_ymin, b2_ymin) intersect_xmin = tf.maximum(b1_xmin, b2_xmin) intersect_ymax = tf.minimum(b1_ymax, b2_ymax) intersect_xmax = tf.minimum(b1_xmax, b2_xmax) intersect_width = tf.maximum(zero, intersect_xmax - intersect_xmin) intersect_height = tf.maximum(zero, intersect_ymax - intersect_ymin) intersect_area = intersect_width * intersect_height union_area = b1_area + b2_area - intersect_area iou = -1 * tf.math.log(tf.math.divide_no_nan(intersect_area, union_area)) return iou
This has been through many, many iterations. As I said, I've solved many other problems with NNs... This is the first one to get me completely stuck. At this point, the network is dramatically stripped down but continues to fail to train at all:
import tensorflow as tf from tensorflow import keras from tensorflow.keras import layers, optimizers tf.keras.backend.set_floatx('float32') # Use Float32s for everything input_shape = x_train_normalized.shape[-3:] model = keras.Sequential() model.add(layers.Conv2D(4, 16, activation = tf.keras.layers.LeakyReLU(alpha=0.2), input_shape=input_shape)) model.add(layers.MaxPooling2D(pool_size=(3, 3), strides=(2, 2))) model.add(layers.Dropout(0.2)) model.add(layers.Flatten()) model.add(layers.Dense(200, activation = tf.keras.layers.LeakyReLU(alpha=0.2))) model.add(layers.Dense(64, activation=tf.keras.layers.LeakyReLU(alpha=0.2))) model.add(layers.Dense(4, activation="sigmoid")) model.compile(loss = iou_loss, optimizer = "adadelta", metrics=['accuracy']) history = model.fit(x_train_normalized, y_train_scaled, epochs=8, batch_size=100, validation_split=0.4)
All pointers are welcome! In the meantime I'm implementing a center point loss function to see if that helps at all.