1
$\begingroup$

Information

I'm working on a binary classification task and used BERT model from transformers library to do it using the custom model below:

class BERT(nn.Module):
  def __init__(self):
    super(BERT, self).__init__()
    self.bert = BertModel.from_pretrained(BERT_PATH, return_dict=False)
    self.dropout = nn.Dropout(0.2)
    self.out = nn.Linear(768, 1)

  def forward(self, ids, mask, token_type_ids):
    outputs = self.bert(ids, attention_mask=mask,token_type_ids=token_type_ids)
    
    # Use the pooled output
    output = self.dropout(outputs[1]) 
    return self.out(output)

What I'm looking for?

Now I'm looking to use a CNN layer on top of BERT with the following configurations to see how my model will perform:

self.cnn = nn.Sequential(
        nn.Conv2d(? ? ?),
        nn.ReLU(),
        nn.MaxPool2d(? ? ?)
    )

The problem encountered.

I have already tried but encountered errors regarding setting the dimensions. In your opinion what configuration should I put in the sequential model to avoid the problem of adjusting the dimensions? If you can copy-paste my code and offer me the final custom model with the right Sequential model included, I will be thankful.

New contributor
noured is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
$\endgroup$

Your Answer

noured is a new contributor. Be nice, and check out our Code of Conduct.

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.