Should I be balancing the data before creating the vocab-to-index dictionary?

Question

My question is about when to balance training data for sentiment analysis.

Upon evaluating my training dataset, which has 3 labels (good, bad, neutral), I noticed there were twice as many neutral labels as the other 2 combined, so I used a function to drop neutral labels randomly.

However, I wasn't sure if I should do this before or after creating the vocab2index mappings.

To explain, I am numericizing my text data by creating a vocabulary of words in the training data and linking them to numbers using enumerate. I think to use that dictionary of vocab2index values to numericise the training data. I also use that same dictionary to numericise the testing data, dropping any words that do not exist in the dictionary.

When I took a class on this, they had balanced the training data AFTER creating the vocab2index dictionary. However, when I thought about this in my own implementation, it did not make sense. What if some words from the original vocabulary are gone completely, then we aren't training the machine learning classifier on those words, but they would not be dropping from the testing data either (since words are dropping from X_test based on whether they are in the vocab2index dictionary).

So should I be balancing the data BEFORE creating the vocab2index dictionary?

I linked the code to create X_train and X_test below in case it help.

def create_X_train(training_data='Sentences_75Agree_csv.csv'):
    data_csv = pd.read_csv(filepath_or_buffer=training_data, sep='.@', header=None, names=['sentence','sentiment'], engine='python')
    list_data = []
    for index, row in data_csv.iterrows():
        dictionary_data = {}
        dictionary_data['message_body'] = row['sentence']
        if row['sentiment'] == 'positive':
             dictionary_data['sentiment'] = 2
        elif row['sentiment'] == 'negative':
             dictionary_data['sentiment'] = 0
        else:
             dictionary_data['sentiment'] = 1 # For neutral sentiment
        list_data.append(dictionary_data)
    dictionary_data = {}
    dictionary_data['data'] = list_data
    messages = [sentence['message_body'] for sentence in dictionary_data['data']]
    sentiments = [sentence['sentiment'] for sentence in dictionary_data['data']]

    tokenized = [preprocess(sentence) for sentence in messages]
    bow = Counter([word for sentence in tokenized for word in sentence]) 
    freqs = {key: value/len(tokenized) for key, value in bow.items()} #keys are the words in the vocab, values are the count of those words

    # Removing 5 most common words from data
    high_cutoff = 5
    K_most_common = [x[0] for x in bow.most_common(high_cutoff)] 
    filtered_words = [word for word in freqs if word not in K_most_common]

    # Create vocab2index dictionary:
    vocab = {word: i for i, word in enumerate(filtered_words, 1)}
    id2vocab = {i: word for word, i in vocab.items()}
    filtered = [[word for word in sentence if word in vocab] for sentence in tokenized] 

    # Balancing training data due to large number of neutral sentences
    balanced = {'messages': [], 'sentiments':[]}
    n_neutral = sum(1 for each in sentiments if each == 1)
    N_examples = len(sentiments)
    # print(n_neutral/N_examples)
    keep_prob = (N_examples - n_neutral)/2/n_neutral
    # print(keep_prob)
    for idx, sentiment in enumerate(sentiments):
        message = filtered[idx]
        if len(message) == 0:
            # skip this sentence because it has length 0
            continue
        elif sentiment != 1 or random.random() < keep_prob:
            balanced['messages'].append(message)
            balanced['sentiments'].append(sentiment)

    token_ids = [[vocab[word] for word in message] for message in balanced['messages']]
    sentiments_balanced = balanced['sentiments']

    # Unit test:
    unique, counts = np.unique(sentiments_balanced, return_counts=True)
    print(np.asarray((unique, counts)).T)
    print(np.mean(sentiments_balanced))
    ##################

    # Left padding and truncating to the same length 
    X_train = token_ids
    for i, sentence in enumerate(X_train):
        if len(sentence) <=30:
            X_train[i] = ((30-len(sentence)) * [0] + sentence)
        elif len(sentence) > 30:
            X_train[i] = sentence[:30]
    return vocab, X_train, sentiments_balanced

def create_X_test(test_sentences, vocab):
    tokenized = [preprocess(sentence) for sentence in test_sentences]
    filtered = [[word for word in sentence if word in vocab] for sentence in tokenized] # X_test filtered to only words in training vocab
    # Alternate method with functional programming:
    # filtered = [list(filter(lambda a: a in vocab, sentence)) for sentence in tokenized]
    token_ids = [[vocab[word] for word in sentence] for sentence in filtered] # Numericise data

    # Remove short sentences in X_test
    token_ids_filtered = [sentence for sentence in token_ids if len(sentence)>10]
    X_test = token_ids_filtered
    for i, sentence in enumerate(X_test):
        if len(sentence) <=30:
            X_test[i] = ((30-len(sentence)) * [0] + sentence)
        elif len(sentence) > 30:
            X_test[i] = sentence[:30]
    return X_test

Answer 1

If you look at the words in your dictionary (vocab) before/after pruning, most likely you'd see there isn't a lot of difference, not so much to affect your model performance.

In fact, creating a dictionary and model training are two more or less indpendent processes. To make your life easier, you could find the largest dev set you can find for building your vocab (excluding test set), and freeze it for all subsequent ETL/modeling. This way you don't have to deal with dictionary versioning, for example, after choosing different subsets of your training data.

Also if you have compute capacity, I'd suggest to upsample positive/negative classes instead, because those neutral samples you're dropping do have signal on the use of language and which borderline ambiguous samples should be treated as neutral.