My answer assumes your fine-tuning architecture simply stacks a single fully-connected layer on top of the BERT [CLS] output, as in Figure 4b of the BERT paper.
Generally, when working with mixed data such as continuous and categorical features, the first step is to simply concatenate all the inputs into one long vector. In your case, you would concatenate a one-hot encoding of speaker ID to the BERT [CLS] output for each example. If you're using tensorflow, you'll need to use the functional API to create a multiple input model, as I outline below.
import tensorflow.keras as keras
# bert_input is the wordpiece tokenized input text
# BERT is the BERT model, such as a tensorflow_hub.KerasLayer with a pretrained model from tfhub.dev
bert_out = BERT(bert_input)
# spkr_input is the one-hot speaker ID encoding.
spkr_input = keras.Input(shape=(num_speakers,), name='spkr_input')
dense_input = keras.layers.concatenate([bert_out, spkr_input])
scores = keras.layers.Dense(num_classes, activation='softmax')(dense_input)
model = keras.Model(inputs=[bert_input , spkr_input], outputs=[scores])
You could also try feeding the one-hot encoded speaker ID vectors through a separate fully-connected network first to obtain a continuous representation (i.e. a speaker embedding) and then concatenate that to the BERT [CLS] output and feed the result into your classification layer. Modifying my example above,
spkr_emb = keras.layers.Dense(spkr_emb_size, activation="relu")(spkr_input)
dense_input = keras.layers.concatenate([bert_out, spkr_emb])
You can find a more detailed guide to mixed input models with Keras at https://www.pyimagesearch.com/2019/02/04/keras-multiple-inputs-and-mixed-data/
