I'm working on a binary classification task and used BERT model from transformers library to do it using the custom model below:
class BERT(nn.Module): def __init__(self): super(BERT, self).__init__() self.bert = BertModel.from_pretrained(BERT_PATH, return_dict=False) self.dropout = nn.Dropout(0.2) self.out = nn.Linear(768, 1) def forward(self, ids, mask, token_type_ids): outputs = self.bert(ids, attention_mask=mask,token_type_ids=token_type_ids) # Use the pooled output output = self.dropout(outputs) return self.out(output)
What I'm looking for?
Now I'm looking to use a
CNN layer on top of
BERT with the following configurations to see how my model will perform:
self.cnn = nn.Sequential( nn.Conv2d(? ? ?), nn.ReLU(), nn.MaxPool2d(? ? ?) )
The problem encountered.
I have already tried but encountered errors regarding setting the dimensions. In your opinion what configuration should I put in the sequential model to avoid the problem of adjusting the dimensions? If you can copy-paste my code and offer me the final custom model with the right Sequential model included, I will be thankful.