1
$\begingroup$

My question is about when to balance training data for sentiment analysis.

Upon evaluating my training dataset, which has 3 labels (good, bad, neutral), I noticed there were twice as many neutral labels as the other 2 combined, so I used a function to drop neutral labels randomly.

However, I wasn't sure if I should do this before or after creating the vocab2index mappings.

To explain, I am numericizing my text data by creating a vocabulary of words in the training data and linking them to numbers using enumerate. I think to use that dictionary of vocab2index values to numericise the training data. I also use that same dictionary to numericise the testing data, dropping any words that do not exist in the dictionary.

When I took a class on this, they had balanced the training data AFTER creating the vocab2index dictionary. However, when I thought about this in my own implementation, it did not make sense. What if some words from the original vocabulary are gone completely, then we aren't training the machine learning classifier on those words, but they would not be dropping from the testing data either (since words are dropping from X_test based on whether they are in the vocab2index dictionary).

So should I be balancing the data BEFORE creating the vocab2index dictionary?

I linked the code to create X_train and X_test below in case it help.

def create_X_train(training_data='Sentences_75Agree_csv.csv'):
    data_csv = pd.read_csv(filepath_or_buffer=training_data, sep='.@', header=None, names=['sentence','sentiment'], engine='python')
    list_data = []
    for index, row in data_csv.iterrows():
        dictionary_data = {}
        dictionary_data['message_body'] = row['sentence']
        if row['sentiment'] == 'positive':
             dictionary_data['sentiment'] = 2
        elif row['sentiment'] == 'negative':
             dictionary_data['sentiment'] = 0
        else:
             dictionary_data['sentiment'] = 1 # For neutral sentiment
        list_data.append(dictionary_data)
    dictionary_data = {}
    dictionary_data['data'] = list_data
    messages = [sentence['message_body'] for sentence in dictionary_data['data']]
    sentiments = [sentence['sentiment'] for sentence in dictionary_data['data']]

    tokenized = [preprocess(sentence) for sentence in messages]
    bow = Counter([word for sentence in tokenized for word in sentence]) 
    freqs = {key: value/len(tokenized) for key, value in bow.items()} #keys are the words in the vocab, values are the count of those words

    # Removing 5 most common words from data
    high_cutoff = 5
    K_most_common = [x[0] for x in bow.most_common(high_cutoff)] 
    filtered_words = [word for word in freqs if word not in K_most_common]

    # Create vocab2index dictionary:
    vocab = {word: i for i, word in enumerate(filtered_words, 1)}
    id2vocab = {i: word for word, i in vocab.items()}
    filtered = [[word for word in sentence if word in vocab] for sentence in tokenized] 

    # Balancing training data due to large number of neutral sentences
    balanced = {'messages': [], 'sentiments':[]}
    n_neutral = sum(1 for each in sentiments if each == 1)
    N_examples = len(sentiments)
    # print(n_neutral/N_examples)
    keep_prob = (N_examples - n_neutral)/2/n_neutral
    # print(keep_prob)
    for idx, sentiment in enumerate(sentiments):
        message = filtered[idx]
        if len(message) == 0:
            # skip this sentence because it has length 0
            continue
        elif sentiment != 1 or random.random() < keep_prob:
            balanced['messages'].append(message)
            balanced['sentiments'].append(sentiment)

    token_ids = [[vocab[word] for word in message] for message in balanced['messages']]
    sentiments_balanced = balanced['sentiments']

    # Unit test:
    unique, counts = np.unique(sentiments_balanced, return_counts=True)
    print(np.asarray((unique, counts)).T)
    print(np.mean(sentiments_balanced))
    ##################

    # Left padding and truncating to the same length 
    X_train = token_ids
    for i, sentence in enumerate(X_train):
        if len(sentence) <=30:
            X_train[i] = ((30-len(sentence)) * [0] + sentence)
        elif len(sentence) > 30:
            X_train[i] = sentence[:30]
    return vocab, X_train, sentiments_balanced
def create_X_test(test_sentences, vocab):
    tokenized = [preprocess(sentence) for sentence in test_sentences]
    filtered = [[word for word in sentence if word in vocab] for sentence in tokenized] # X_test filtered to only words in training vocab
    # Alternate method with functional programming:
    # filtered = [list(filter(lambda a: a in vocab, sentence)) for sentence in tokenized]
    token_ids = [[vocab[word] for word in sentence] for sentence in filtered] # Numericise data

    # Remove short sentences in X_test
    token_ids_filtered = [sentence for sentence in token_ids if len(sentence)>10]
    X_test = token_ids_filtered
    for i, sentence in enumerate(X_test):
        if len(sentence) <=30:
            X_test[i] = ((30-len(sentence)) * [0] + sentence)
        elif len(sentence) > 30:
            X_test[i] = sentence[:30]
    return X_test
$\endgroup$
0
$\begingroup$

If you look at the words in your dictionary (vocab) before/after pruning, most likely you'd see there isn't a lot of difference, not so much to affect your model performance.

In fact, creating a dictionary and model training are two more or less indpendent processes. To make your life easier, you could find the largest dev set you can find for building your vocab (excluding test set), and freeze it for all subsequent ETL/modeling. This way you don't have to deal with dictionary versioning, for example, after choosing different subsets of your training data.

Also if you have compute capacity, I'd suggest to upsample positive/negative classes instead, because those neutral samples you're dropping do have signal on the use of language and which borderline ambiguous samples should be treated as neutral.

$\endgroup$
7
  • $\begingroup$ Thanks for this response! I never thought about just finding a large dev set but that makes a lot of sense. Could you explain what you mean by upsampling though? Thanks! $\endgroup$ – Nore Apr 7 '20 at 9:56
  • $\begingroup$ Sure. So say your class distribution is pos=5, neg=5, neutral=10, you do sampling with replacement from pos and neg until you have 10 samples for all classes. In pandas you can easily do this with df.sample and if you do a bit of search you'll find other ways/libs to help you with that. $\endgroup$ – olix20 Apr 7 '20 at 10:30
  • $\begingroup$ Ah okay, so this would create duplicate sentences in the training correct? Have you also heard of the class_weights parameter of most sklearn models? It seems to balance classes by assigning weights. Someone else pointed it out to me and I think it does what I want but much nicer. Wanted to ask if you knew it as well? Thanks! $\endgroup$ – Nore Apr 7 '20 at 10:32
  • $\begingroup$ yes correct. Indeed an alternative is class_weights, but depends if the model you want to use implements it. $\endgroup$ – olix20 Apr 7 '20 at 12:05
  • $\begingroup$ Hey I had a further ponder over finding a large dev set but something occurred to me. Wouldn't this encode various words that exist in the dev set and testing set but not in the training set? I'm wondering what the point would be of keeping those words in the test set if the classifier isn't even trained on them. At the moment, when I use BoW I drop the words not in the training set from the test set. What's the intuition behind keeping them using a dev set? Would they just be ignored by the classifier anyway? Or rather, how do trained classifiers deal with those kinds of words? Thanks! $\endgroup$ – Nore Apr 9 '20 at 7:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.