2
$\begingroup$

I have a neural network that should be able to classify documents to target label A. The problem is that the network is actually classifying label B, which is an easier task.

To make the problem more clear: I need to classify documents from different sources. In the training data each source occurs repeatedly, but the network should be able to work on unknown sources. All documents from a single source have the same class. In this case, it is easier to identify sources than the target label so in practice the network is not really identifying the target label, but the source.

The solution to this problem is making sure that the model is bad at identifying the sources in the training data, while still attaching the right target labels.

I think the first step is to get two output layers, one for the target label and one for identifying which source it is from. My approach fails however at the training procedure: I want to minimize the loss on the target output, but maximize the loss on the non-target output. But if I maximize the loss on that non-target output, that does not mean that the network 'unlearns' the non-target labels. So the main question for the non-target output is:

TLDR; How do I define a training procedure that minimizes the loss on a non-target output layer, and then maximizes that loss on all layers before it. My goal is to have a network that is good at classifying label A, but bad at a related label B. If anyone wants to give a code example, my prefered framework is PyTorch.

$\endgroup$
0
$\begingroup$

You can maximize the loss by doing 1-loss (if the range of the loss is between 1 and 0. However, I am not sure if that would help. The network can just propagate the weights of the source classifier to always output the wrong answer. a better approach is to hand craft features of the document to make it less identifiable as the source. If the document have stated the source, remove it. This may help. Also, another way is to use documents from more sources so the network cannot use the source to classify the label. Hope I can help you.

$\endgroup$
0
$\begingroup$

A little more information about the documents will be helpful. I am guessing that your scenario has webpages from different websites, you're feeding html pages to the network and the page contains the website name or url, which the network is picking up on and using it to label. I am assuming you're using a RNN or similar network for the classification task.

Your question is to identify when your network is using the source identifying feature (name or url), and instruct it to not use this feature. If you can do some preprocessing and identify where in the document the source name/url is, you may tag each word/character in the input with this information. You may use this tag to add a large regularizing loss term which brings down the weights connected to the tagged input words/characters in the first layer.

However, a better strategy will be to remove all instance of the header / url which contains website info, as @clement-hui suggested. This is cleaner and easier to implement than above.

Perhaps your network is identifying the document source slightly more indirectly, and above preprocessing is not sufficient/useful. In this case, you may want to make random chunks of your document, and give each chunk the label corresponding to that document. Here it is more likely that some chunks will not have the source identifying information, and the network will be forced to pick a strategy which labels using the content rather than the source.

At test time you may either feed the whole document or random chunks from the document and use majority voting to get the final answer.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.