How do I know if image after image enhancement is better than before? (Image Preprocessing)

Question

There are few common image enhancement:

1. brightness -> r = s + b
2. negative -> s = 255 - r
3. contrast -> scretching (flexible) dan thresholding (binary image)
4. smoothing -> generate blur
    penapis non linear (min, max, median)
5. sharpening (high pass filter), suit for getting edge

How do I know what should I enchance? Is it brightness, contrast, or what?

And how do I know if image is better after enhancement or not (evaluating result) objectively instead of subjectively by our perspective? Mainly for medical imaging such as USG, CT-SCAN, etc...

As far As I know, I can see the histogram, but I still don't know what histogram to evaluate the result.

Answer 1

"Better" has a very wide definition. Is it faster? Better accuracy? Better precision? First, you need to define what you mean by "Better"

In general, it depends on your task (mainly) and your model. Some image processing methods will give benefit in some cases, but they also degrade performance in other cases (e.g. changing rgb image to b/w). I suggest two approaches:

1. The easiest one is to use it in the downstream task, which means your main task, and see the evaluation result. For example, change the image brightness and see the accuracy of your model, whether it helps or not. With this approach, you need to do several trial-error experiments

2. If you don't want to use the downstream task, you can read the previous research or understand your problem domain and model behavior. It could give you some information (or intuition) about what's good and not. For example, a quick search leads me to this research that shows some techniques for data augmentation affect the model performance for MRI.

Answer 2

As said in other answers, it can be very difficult to quantify "better" and this depends on your problem domain. However, we can quantify "different" (with respect to the original image). This may be able to tell you how much of an effect your changes may have step-by-step.

A common metric for this is the structural similarity index, which is particularly concerned with image texture. Although regular (Shannon's) entropy isn't suitable for images, Larkin (2016) proposes a modification, dubbed "delentropy" for this task. I use an implementation from this gist, which I've condensed make it into a easy to use function:

def delentropy(image):
    # Using a 2x2 difference kernel [[-1,+1],[-1,+1]] results in artifacts!
    # In tests the deldensity seemed to follow a diagonal because of the
    # assymetry introduced by the backward/forward difference
    # the central difference correspond to a convolution kernel of
    # [[-1,0,1],[-1,0,1],[-1,0,1]] and its transposed, produces a symmetric
    # deldensity for random noise.
    if True:
        # see paper eq. (4)
        fx = ( image[:,2:] - image[:,:-2] )[1:-1,:]
        fy = ( image[2:,:] - image[:-2,:] )[:,1:-1]
    else:
        # throw away last row, because it seems to show some artifacts which it shouldn't really
        # Cleaning this up does not seem to work
        kernelDiffY = np.array( [ [-1,-1],[1,1] ] )
        fx = signal.fftconvolve( image, kernelDiffY.T ).astype( image.dtype )[:-1,:-1]
        fy = signal.fftconvolve( image, kernelDiffY   ).astype( image.dtype )[:-1,:-1]
    diffRange = np.max( [ np.abs( fx.min() ), np.abs( fx.max() ), np.abs( fy.min() ), np.abs( fy.max() ) ] )
    if diffRange >= 200   and diffRange <= 255  : diffRange = 255
    if diffRange >= 60000 and diffRange <= 65535: diffRange = 65535

    # see paper eq. (17)
    # The bin edges must be integers, that's why the number of bins and range depends on each other
    nBins = min( 1024, 2*diffRange+1 )
    if image.dtype == float:
        nBins = 1024
    # Centering the bins is necessary because else all value will lie on
    # the bin edges thereby leading to assymetric artifacts
    dbin = 0 if image.dtype == float else 0.5
    r = diffRange + dbin
    delDensity, xedges, yedges = np.histogram2d( fx.flatten(), fy.flatten(), bins = nBins, range = [ [-r,r], [-r,r] ] )
    if nBins == 2*diffRange+1:
        assert( xedges[1] - xedges[0] == 1.0 )
        assert( yedges[1] - yedges[0] == 1.0 )

    # Normalization for entropy calculation. np.sum( H ) should be ( imageWidth-1 )*( imageHeight-1 )
    # The -1 stems from the lost pixels when calculating the gradients with non-periodic boundary conditions
    #assert( np.product( np.array( image.shape ) - 1 ) == np.sum( delDensity ) )
    delDensity = delDensity / np.sum( delDensity ) # see paper eq. (17)
    delDensity = delDensity.T
    # "The entropy is a sum of terms of the form p log(p). When p=0 you instead use the limiting value (as p approaches 0 from above), which is 0."
    # The 0.5 factor is discussed in the paper chapter "4.3 Papoulis generalized sampling halves the delentropy"
    H = - 0.5 * np.sum( delDensity[ delDensity.nonzero() ] * np.log2( delDensity[ delDensity.nonzero() ] ) ) # see paper eq. (16)
    return H

In any case, I'll reiterate that downstream performance metrics are the way to quantify "better," but inspecting the effect of each filter will help guide you on what changes are could be impactful and which may not be.

Answer 3

I've worked on color enhancement and your question is very valid. In practice you have human experts qualitatively evaluating the algorithms. The other option is to use quantitative evaluation that is what your question points to. Quantitative evaluation depends on the task, as image enhancement includes denoising, color enhancement, sharpening, superesolution and more.

For color enhancement researchers use datasets that have i) pairs of images before and after enhancement (e.g. Adobe-MIT5K dataset) or ii) good images or bad images (can involve ratings). In NeurIPS 2022 a dataset involving edited images and natural language comments was introduced (link).

For the first case in which you have a target image the metrics are usually structural similarity, PSNR and norms in different colorspaces (e.g. CIELab).

There are many methods that do image enhancement and the main difficulty is to evaluate them. Once you have a way to automatically evaluate them you can probably optimize for that criteria. Don't forget that it's highly subjective, it's been shown that in average people in tropical regions (e.g. china) prefer more contrasted and vibrant colors that those in northern Europe.