Each instance has multiple labels at a time. Now you can see I have 2 instances for Label2, however, only one, one instance for other labels. It means, we have class imbalanced issues. I read about the term "Adding Class Weights for an imbalanced dataset", however, I could not understand, how actually it works and beneficial. Can anyone explain this method generally as well as according to my given toy data set. Other than that, how we handle this Miss Lable (nan)?
The paper A systematic study of the class imbalance problem in convolutional neural networks is a great overview on class imbalance approaches. Section 2 summarizes various methods commonly used. They categorize "Adding Class Weights for an imbalanced dataset" under the technique "Cost sensitive learning":
Cost sensitive learning. This method assigns different cost to misclassification of examples from different classes . With respect to neural networks it can be implemented in various ways. One approach is threshold moving  or post scaling  that is applied in the inference phase after the classifier is already trained. Similar strategy is to adapt the output of the network and also use it in the backward pass of backpropagation algorithm . Another adaptation of neural network to be cost sensitive is to modify the learning rate such that higher cost examples contribute more to the update of weights. And finally we can train the network by minimizing the misclassification cost instead of standard loss function
Without further context, "Adding Class Weights for an imbalanced dataset" can mean many things as enumerated by the above. But if I had to guess, the most common meaning is that they weigh the misclassification cost differently per label by multiplying it with a different weight variable.
For example, maybe label 1, 2, and 3, when misclassified, gets a 1x multiplier in the training loss (the standard weight), but label 4 gets a 3x multiplier because it is roughly 3x more important, and label 5 gets a 0.5x multiplier because it is 0.5x less important.
Here is an example of how to do this with Keras.
(FYI the paper above recommends you over-sample the minority class rather than using the cost-sensitive learning approach to help with class imbalance)
As for missing labels, the quickest solution is just to skip those training instances when training for that particular missing label. If you want to still use the training instances somehow, a common approach is to fix the missing label with semi-supervised learning approaches.
One way to do semi-supervised learning is self-training, where your model is trained on only the labeled instances, and then makes predictions on the missing labels. High-confidence predictions of the missing labels are then added to the training data, and the model is trained on the new training data. This process repeats until convergence of the training data.