# Can mini-batches for stochastic gradient be balanced but not representative of the training data?

When we construct mini-batches for stochastic gradient, it is important to ensure that the different mini-batches are balanced (for example, in case of classification they contain the same ratio for each class) and representative for the whole dataset.

Can mini-batches be balanced but not representative? Any example?

The problem is, what mathematical definition do you give to "balanced" and "representative"?

Balanced usually means that classes are uniformly distributed, but then you are screwing up your prior, but that's nothing that an importance sampling coefficient might not fix

However, the representative part is ill defined. By itself, you want enough data such that: $$\nabla L^*(\theta)^T \nabla L(\theta) \ge 0$$ $$L$$ being the stochastic loss, and $$L^*$$ the actual "full" loss... in other words, you want that your stochastic gradient is pointing "pretty much" in the same direction as the full gradient.

Now, how do we know that this is true? we don't... however, the more "stochastic" it is, the more likely is this to happen (thus increasing batch sizes usually fixes this)

So, can a a batch of data be balanced and be not representative? Yes, you can pretty easily build an example where this is true

Since it seems that the words were not enough, here you are an example

Consider the following setting (dotted line is the decision boundary)

If you sample the two RED outliers, the gradient from that minibatch will still drive the decision boundary towards the optimal one (hope we agree on this point), thus

$$\nabla L^\text{all}(\theta)^T \nabla L^\text{minibatch RED}(\theta) \ge 0$$

However, now consider the following decision boundary

Clearly, the loss induced by this decision boundary, is much lower by the previous one... however, now if you sample the RED minibatch, the gradient will say "nope, totally disagree, you should completely flip the prediction since all O are predicted as X and all X are predicted as O", in other words: $$\nabla L^\text{all}(\theta)^T \nabla L^\text{minibatch RED}(\theta) \le 0$$

Note: in all cases the ratio between O/X in the whole dataset and in the minibatch is preserved (1:1 ratio), thus the minibatch is indeed balanced, yet it's not representative

• @DSPinfinity From a statistical perspective, a sample (mini-batch) of a population (dataset) is representative if its statistics are pretty much the same of the population. Let's say you have a dataset $X$ with 1 feature, with mean $\mu_X=1$. Now you're mini-batches should have approximately the same mean value, therefore not something very different like $2$ or $-1$, for example. Anyway, I think that the important part is that the samples are representative if they follow the same distribution of the dataset. So balancing classes makes the "class" distribution uniform, without modes (peaks). Nov 18, 2023 at 10:45
• @LucaAnzalone No, consider logistic regression, take 2 linearly separable classes, and add one "outlier" per class in the opposite cluster. Now, if your minibatch is composed by only these 2 outlier points, your batch is still balanced, but it's not representative, as the direction of the update will be almost opposite to the true gradient (and (1) your minibatch can be seen as a single sample MC estimate of your parameters (2) these two points might definitely come from the exact distribution of your data). Nov 19, 2023 at 0:34
• @LucaAnzalone To conclude, it makes little sense to define "representativeness" as closeness in the distribution statistics space, because then you should define a measure of closeness, and some threshold over which is considered "not representative", where instead the definition with the gradient has no "subjectivity" innit, and as said earlier, you can most likely have a non representative sample, and it gets more likely the closer you get to your local minima Nov 19, 2023 at 0:36
• @LucaAnzalone added some illustrations on the answer, hope it helps Nov 19, 2023 at 0:46
• @DSPinfinity added some illustrations on the answer, hope it helps Nov 19, 2023 at 0:46

Though you've already accepted an answer, from your comments it seems you're still somewhat unclear and need an example. Take your example of (binary) classification, usually balanced training data means you have more or less equal number of samples for each class label. Any due to the intrinsic random nature of each mini-batch during gradient descent (MBGD), it's very likely some mini-batches data will not be representative in the sense that it may become imbalanced. But so long as MBGD converges, it's generally converge faster and more computationally efficient leveraging parallelism than full batch GD, the impact of possibly many imbalanced mini-batches is typically minimized as the model updates its parameters based on the entire dataset and over multiple epochs.

Here's a simple binary logistic classification example for a balanced dataset with two class labels of equal number of samples (50 for each) and two features, code is provided at then end. Total dataset has 100 samples, mini-batch size is set to 5, and epoch is tuned to 1000. As you can see from the output many mini-batches are not representative for the whole dataset (many have 0%, 20%, 60%, 80% class_1 data), but the final result is still good and close to the full batch result as you can easily verify.

output result:
i=0, minibatch_size=5 class_1%=0.6
i=5, minibatch_size=5 class_1%=0.4
i=10, minibatch_size=5 class_1%=0.4
i=15, minibatch_size=5 class_1%=0.8
i=20, minibatch_size=5 class_1%=1.0
i=25, minibatch_size=5 class_1%=0.6
i=30, minibatch_size=5 class_1%=0.0
i=35, minibatch_size=5 class_1%=0.4
i=40, minibatch_size=5 class_1%=0.8
i=45, minibatch_size=5 class_1%=0.2
i=50, minibatch_size=5 class_1%=0.4
i=55, minibatch_size=5 class_1%=0.6
i=60, minibatch_size=5 class_1%=0.6
i=65, minibatch_size=5 class_1%=0.8
i=70, minibatch_size=5 class_1%=0.8
i=75, minibatch_size=5 class_1%=0.6
i=80, minibatch_size=5 class_1%=0.4
i=85, minibatch_size=5 class_1%=0.2
i=90, minibatch_size=5 class_1%=0.2
i=95, minibatch_size=5 class_1%=0.2
i=0, minibatch_size=5 class_1%=0.6
i=5, minibatch_size=5 class_1%=0.2
i=10, minibatch_size=5 class_1%=0.2
i=15, minibatch_size=5 class_1%=0.8
i=20, minibatch_size=5 class_1%=0.4
...
i=95, minibatch_size=5 class_1%=0.2
----
import numpy as np
import matplotlib.pyplot as plt

np.random.seed(42)

# Class 0: Generate points in a cluster around (-1, -1)
num_class_0 = 50
X_class_0 = np.random.randn(num_class_0, 2) - 1 * np.ones((num_class_0, 2))
y_class_0 = np.zeros((num_class_0,))

# Class 1: Generate points in a cluster around (1, 1)
num_class_1 = 50
X_class_1 = np.random.randn(num_class_1, 2) + 1 * np.ones((num_class_1, 2))
y_class_1 = np.ones((num_class_1,))

X = np.concatenate((X_class_0, X_class_1), axis=0)
X_b = np.c_[np.ones((X.shape[0], 1)), X]
y = np.concatenate((y_class_0, y_class_1))

# Sigmoid function for logistic regression
def sigmoid(z):
return 1 / (1 + np.exp(-z))

# Initialize parameters for logistic regression (bias and weights)
theta = np.random.randn(X_b.shape[1], 1)

# Define hyperparameters
learning_rate = 0.2
batch_size = 5
n_iterations = 1000
m = len(X)
for iteration in range(n_iterations):
shuffled_indices = np.random.permutation(m)
X_shuffled = X_b[shuffled_indices]
y_shuffled = y[shuffled_indices].reshape(-1, 1)

for i in range(0, m, batch_size):
xi = X_shuffled[i:i+batch_size]
yi = y_shuffled[i:i+batch_size]
print(f'i={i}, minibatch_size={np.size(yi)} class_1%={np.sum(yi)/np.size(yi)}')
gradients = 1/batch_size * xi.T.dot(sigmoid(xi.dot(theta)) - yi)

# Function to predict classes
def predict(X):
return (sigmoid(X.dot(theta)) >= 0.5).astype(int)

# Plotting the decision boundary
x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
xx, yy = np.meshgrid(np.linspace(x_min, x_max, 1000), np.linspace(y_min, y_max, 1000))
X_new = np.c_[np.ones((len(xx.ravel()), 1)), xx.ravel(), yy.ravel()]
Z = predict(X_new).reshape(xx.shape)
print(f'theta={theta}')

plt.contourf(xx, yy, Z, alpha=0.3)
plt.scatter(X_class_0[:, 0], X_class_0[:, 1], label='Class 0')
plt.scatter(X_class_1[:, 0], X_class_1[:, 1], label='Class 1')
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')
plt.title('MBGD Logistic Regression on Imbalanced Data')
plt.legend()
plt.show()