2
$\begingroup$

Let's say we have several vector points. My goal is to distinguish the vectors, so I want to make them far from each other. Some of them are already far from each other, but some of them can be positioned very closely.

I want to get a certain mapping function that can separate such points that are close to each other, while still preserving the points that are already far away from each other.

I do not care what is the form of the mapping. Since the mapping will be employed as pre-processing, it does not have to be differentiable or even continuous.

I think this problem is somewhat similar to 'minimizing the maximum distance ratio between the points'. Maybe this problem can be understood as stretching the crushed graph to a sphere-like isotropic graph.

I googled it for an hour, but it seems that the people are usually interested in selecting the points that have such nice characteristics from a bunch of data, rather than mapping an existing vector points to a better one.

So, in conclusion, I could not find anything useful.

Maybe you can think 'the neural network will naturally learn it while solving classification problem'. But it failed. Because it is already struggling with too many burdens. So, this is why I want to help my network with pre-processing.

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ isn't it what kernels do? $\endgroup$
    – Alireza
    May 20, 2020 at 20:43
  • 1
    $\begingroup$ which kernel do you mean? $\endgroup$
    – Jun
    May 21, 2020 at 3:54
  • 1
    $\begingroup$ The general concept of kernel trick in kernel methods in machine learning. such as SVM. the kernels do what you're saying: transforming to a new hyperspace which a discriminative hyperplane can easier separate classes. $\endgroup$
    – Alireza
    May 21, 2020 at 7:58
  • 1
    $\begingroup$ this is interesting, but, why do you want to do this? $\endgroup$
    – carlo
    Jun 19, 2020 at 16:58
  • 1
    $\begingroup$ "vector points" means "set of points described as vectors" ? Is a 1D, 2D, 3D, ... problem ? The maximum distance will be reached send each point to the infinite (except one of them at origin). Do you have restrictions for the base space ? $\endgroup$
    – pasaba por aqui
    Oct 17, 2020 at 17:59

1 Answer 1

Reset to default
1
$\begingroup$

An interesting question. I would start by finding n nearest neighbors of each data point, then calculate their center of mass c and the point's distance d to its nth nearest neighbor. The smaller the d is the larger the density is around a given point. You could then iteratively step every point away from their c in an inverse proportion to the distance d with a suitable step size. This would spread out the clusters.

But this won't help you transform any new points in the dataset, maybe you can learn this arbitrary mapping R^n -> R^n by using an other neural network and apply it to new samples?

This is the first ad-hoc idea which came to my mind. It would be interesting to see a 2D animation of this.

A more rigorous approach might be a variational autoencoder, you can embed the data in a lower dimensional space with approximately normal distribution. But it doesn't guarantee that clusters would be as spread out as you'd like. An alternative loss function would help with that, for example every point's distance to their original nth closest neighbor should be as close to one as possible.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ Thanks for your advise. I also considered about using k-nearest neighbor(kNN) based scheme, but as you know it includes max (or min) function in it. So I think it is very hard to use kNN related metric as a loss function to train VAE, even though the goal mapping itself do not have to be a differentiable one. But maybe some relaxed version of it, such as stochastic NN could be a naive replacement. $\endgroup$
    – Jun
    May 21, 2020 at 3:58
  • $\begingroup$ After studying neural networks more I recently came up with a new idea (might not be novel): train an autoencoder (it doesn't matter if the encoded "code" dimension is smaller or same size as the input) and add extra penalty to the "code" activations if they are too close on known dissimilar inputs. I am about to write a blog article about this at some point. $\endgroup$
    – NikoNyrh
    Jan 5, 2022 at 21:28
  • $\begingroup$ Too late to edit the previous comment. Surely this is not a novel idea, it combines an autoencoder with a contrastive loss. But wikipedia's article on contractive autoencoder mentions only positive samples, which makes it more of a denoising autoencoder IMO. $\endgroup$
    – NikoNyrh
    Jan 5, 2022 at 21:37

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .