I have a binary classification problem, where a false positive error has a very big cost compared to the false negative error.
Is there a way to design a classifier for such problems (preferably, with an implementation of the algorithm)?
There is no predefined classifier for any problem. Two main features of a classifier is its cost function and its corresponding weight updation formula. Since, your problem statement requires a huge cost for falsely classifying a particular class one approach will be.
J'put together. You can lookup the cost function of a logistic classifier to see how two separate cost functions are merged together here.
alphafor both the classes should be different. The larger learning rate will be for the one which is the more important class, since you don't want to classify it improperly(increasing
alphacompared to the other class will reduce or risk of missclassifying it). Exact learning rate depends from case to case.
Thus, I have tailored the two main features of the classifier to solve this problem:
@DuttaA has pretty much mentioned the two most appropriate approaches to having this facility. Either the penalty of false positives should be high or the learning rate for the correct class should be high.
I'll give two real-life examples to help you understand it better.
Say you have to teach a teen that substance abuse is injurious to health (eg. Frequent smoking is a negative habit). But the teen ends up learning from high effects of the drugs that it is good (false positive) and gets addicted to it. You would strictly want to avoid this kind of a situation (false positive error having a very big cost as compared to false negative error).
In general, to model the situation when the costs are different, we picture a cost matrix. For a two-class classification problem, the cost matrix would look like:
Now, when designing your cost function, you would want to take into account the weight corresponding to each of the situation. A simple python code would be as follows:
def weighted_cost(pred, act): if pred==P and act==P: return C_TP * cost(pred, act) if pred==P and act==N: return C_FP * cost(pred, act) if pred==N and act==P: return C_FN * cost(pred, act) if pred==N and act==N: return C_TN * cost(pred, act)
Where, pred is the predicted class and act is the actual class. Here, C_TP, C_FP, C_TN, C_FN represent the weights of true positive, false positive etc. The cost(pred, act) function will calculate the loss of one training example. You would want to use the weighted_cost function for finding the loss after one training example.
The second approach that @DuttaA mentioned was to vary the learning rate. In real life, you can relate this to the situation when you were asked to write a word multiple times if you forget its spelling so that you remember it better. In a way, you learn the correct spelling of the word.
Here, increasing the value of the learning rate (say 4 x alpha) for a class can be viewed as updating the value of the weights multiple times (4 times) with the old learning rate (alpha), similar to what we do by writing the correct spelling of the word multiple times. So, a more important class (in your case it will be the Negative Class) should be given more alpha, because a false positive (misclassification of the negative class) has a high penalty. You learn to recognize the correct (negative) class by learning it more number of times (as in the case of learning the spelling of the word).
Let me know if you need any further clarification.
A funky way of doing this with less overhead is to just over fit the data up to some degree. The reason is when you try to over fit the data with the classifier the classification bound tends to wrap around the clusters very tightly and with that model you can some times miss classify positive classes as negative(due to high variance) but there are comparatively less situations where you end up miss classifying negative classes as positive. The level of overfitting that needs to be performed is just based on your FP and FN trade off.
I don't think this as a permanent fix but can come handy up to some extent.