I have a dataset of sentences that I embedded using the USE for training an image2Seq model. But when I applied t-SNE to the embeddings, applied K-Means, and plotted a scatterplot, I could see that most of the points are very closely packed, and there are clusters of data. This clustering is making the model predictions biased and not valid for my system.
So is there anything I can do to improve the performance? Any tweaking or trick in the model architecture? Or Any transformation on the points?