Are PreLU and Leaky ReLU better than ReLU in the case of noisy labels?

Question

Let's assume I want to build a semantic segmentation algorithm, based on Multires-UNET. My GT-masks are messy and generated by a GAN, but they are getting better and better over time. The goal is knowledge expansion (based on the paper Noisy-Student).

Can you generally say that PreLU and Leaky Relu are better for noisy labels (or imperfect ones), like the situation in GANs in general?

Answer 1

The provably more-reliable approach to learning with noisy labels/label errors (instead of altering the model architecture or loss function) is to find the label errors and train without them on cleaned data -- a result shown in the confident learning paper (Northcutt, Lu, Chuang; 2021)). I am an author on this paper.

Re-phrasing your question to two questions: (1) "how can I generally find label errors in any dataset for any model?" and (2) "how can I train a model on noisy data as if it was clean data?".

There are one line of code answers to each of these questions using the cleanlab python package which implements the algorithms and theory in the confident learning paper and works for any dataset you can train a classifier on for most data formats, ML and deep learning frameworks, and data modalities, e.g. image, text, tabular, and audio data.

link to python package: https://github.com/cleanlab/cleanlab

Find label issues in 1 line of code

from cleanlab.classification import CleanLearning
from cleanlab.filter import find_label_issues

# Option 1 - works with sklearn-compatible models - just input the data and labels ツ
label_issues_info = CleanLearning(clf=sklearn_compatible_model).find_label_issues(data, labels)

# Option 2 - works with ANY ML model - just input the model's predicted probabilities
ordered_label_issues = find_label_issues(
    labels=labels,
    pred_probs=pred_probs,  # out-of-sample predicted probabilities from any model
    return_indices_ranked_by='self_confidence',
)

Train a model as if the dataset did not have errors -- 3 lines of code

from sklearn.linear_model import LogisticRegression
from cleanlab.classification import CleanLearning

cl = CleanLearning(clf=LogisticRegression())  # any sklearn-compatible classifier
cl.fit(train_data, labels)

# Estimate the predictions you would have gotten if you trained without mislabeled data.
predictions = cl.predict(test_data)

Documentation and runnable tutorials for cleanlab: https://docs.cleanlab.ai/