5 votes
Accepted

How to refine K-means clustering on a data set?

The usual parameters to adjust in a k-means: Number of clusters (recall many clusters can have same label). Distance definition (euclidean is the most basic, Gauss is an improvement) Selection of ...
5 votes

What is graph clustering?

In graph clustering, we want to cluster the nodes of a given graph, such that nodes in the same cluster are highly connected (by edges) and nodes in different clusters are poorly or not connected at ...
  • 36.4k
3 votes
Accepted

How to compute the number of centroids for K-means clustering algorithm given minimal distance?

Yes, the silhouette method (which is implemented in sklearn as silhouette_score) is commonly used to assess the quality of ...
  • 36.4k
3 votes
Accepted

Is unsupervised learning a branch of AI?

There is a problem with confining Artificial Intelligence to a single definition, because it has become an umbrella term encompassing many fields of science. It has come a long way from the "thinking ...
  • 3,133
3 votes

How to define machine learning to cover clustering, classification, and regression?

I report three definitions of machine learning (ML) and I also explain that ML can be divided into multiple sub-tasks or sub-categories in this answer. However, it may not always be clear why ...
  • 36.4k
2 votes

How can I cluster this data frame with several features and observations?

A typical clustering algorithm is k-means (and not k-NN, i.e. k-nearest neighbours, which is primarily used for classification). There are other clustering algorithms, such as hierarchical clustering ...
  • 36.4k
2 votes

How to detect patterns in salary distribution if we are suspecting malicious distribution based on employee's region?

A simple initial approach would be to separate it by position and check for each: Use linear regression: $\hat{salary} = \sum_i \alpha_i * \hat{region}_i + \sum_k \beta_k * \mathbf{1}[\hat{gender}=k]$...
  • 2,329
2 votes

Is this dataset with only two features suitable for clustering with k-means?

One problem with clustering algorithms is that they will typically find you a solution, ie they will split your data set into clusters, but it will find you a structure even if there isn't one. Your ...
  • 5,232
2 votes

Could clustering be used to parse pdf documents to get headings and titles?

Yes, you could use clustering: Encode your features as a feature vector and feed it into a clustering algorithm (see Finding Groups in Data for a comprehensive description of these). You could use ...
  • 5,232
2 votes

Which metric should I use to assess the quality of the clusters?

You can compute Silhouette Coefficient for your aim. Its values mean: 1: Means clusters are well apart from each other and clearly distinguished. 0: Means clusters are indifferent, or we can say that ...
  • 1,703
2 votes
Accepted

What happens if all the features are correlated with each other before clustering?

Essentially, yes. One feature predicts to a reasonably high degree what the other features look like, so the additional features have limited discriminatory power. Obviously there is some effect, as ...
  • 5,232
2 votes
Accepted

How to tackle the human error made in labeling datasets for classification tasks like facial expression recognition?

In general the only way to deal with this is by quantifying these labeling mistakes in the output of the model, since the model will learn for them. And in many cases these are not really mistakes, ...
  • 1,087
2 votes
Accepted

Interpretation of the Dynamic Time Warping (DTW) graph

The graph plots two things: the optimal warping path; the accumulated cost matrix (which looks like a heat map). To interpret the graph, suppose we draw some other path from bottom left to top right....
1 vote

Pre-Trained Model for Occupational Coding

My initial (naive) approach would be to embed the textual data to some representation, using a pre-trained text embedding network. While BERT indeed works, I'd even start off with a simpler embedding,...
1 vote

Deep Clustering Approach for Unsupervised Video Anomaly Detection

What can be a similar criteria that we can use for pseudo labelling if we use a clustering method instead of an Autoencoder? Maybe the amount of Data points inside a Cluster. Like you said, an ...
1 vote
Accepted

Clustering by using Locality sensitive hashing *after* Random projection

I think the following is the way to look at your question. RP reduces dimensionality based on distance. LSH clusters data based on a similar distance method used in RP. The primary function of any ...
1 vote

Feature Engineering on transactional dataset clustering

The average transaction is a central measure, while the minimum and maximum transactions together give an idea of dispersion. However, these can be very sensitive to individual purchases that might ...
1 vote

How do you evaluate a k-medoids cluster model?

In Finding Groups in Data, Kaufman and Rousseeuw describe ways to evaluate the quality of clusterings. If I remember correctly (it's been some time that I worked with this), for k-means algorithms you ...
  • 5,232
1 vote
Accepted

What clustering algorithms work best for datasets with only binary categorical features?

Any clustering algorithm should work -- the main issue is the similarity or distance metric that determines how similar (or different) two elements are. This is often something like Euclidean distance,...
  • 5,232
1 vote

How to handle list features in clustering?

You essentially want to have a numerical value to represent the similarity of the lists of two distinct objects. There are a number of metrics to deal with that, eg the Jaccard Index or Dice's ...
  • 5,232
1 vote

Is there a clustering algorithm that can make n clusters and the n+1 "others" cluster?

So, I've prepared some data that resembles your sketch: ...
  • 1,982
1 vote

Which metric should I use to assess the quality of the clusters?

One more popular metric for this is the Davies Bouldin Score. You can also take a look at the clustering metrics in scikit documentation.
1 vote
Accepted

What exactly is the eigenspace of a graph (in spectral clustering)?

In spectral clustering we not find the eigenvectors of a graph (a graph is not a matrix) but the eigenvalues/eigenvectors of the Laplacian matrix related to the adjacency matrix of the graph: graph =&...
1 vote

How can I cluster this data frame with several features and observations?

Yes you can use KNN algorithm to cluster (well actually its a classification not a clustering if you use KNN) the data. But, first you need to set one feature as a label because KNN is a supervised ...
1 vote

How can I cluster based on the complementary categories?

Note: K-means does not assume an interpretation/label of the clusterings - in fact it is an unsupervised algorithm. The interpretations are a result of human analysis after running K-means. For ...
  • 1,066
1 vote

What is the role of the 'fuzzifier' w in Fuzzy Clustering?

Its not required, you can have $m=1$, actually it can be any number $\geq 1$. Now the better question is why to have it? The answer is that it adds a smoothing effect. Lets look at it in each of ...
  • 2,329
1 vote

How to compute the number of centroids for K-means clustering algorithm given minimal distance?

If you look at Kaufman & Rousseeuw (1990), Finding Groups in Data, they describe an algorithm to evaluate the quality of clusters in agglomerative clustering. You run the clustering algorithm with ...
  • 5,232
1 vote
Accepted

How do we know the classification boundaries of the data?

This is the classic question of what structure is or can be. It relates directly to the concepts of generalization, pattern recognition, over-fitting in surface fitting strategies, and learning tabula ...
1 vote

What techniques to explore for dynamic clustering of documents (emails)?

It sounds like you are trying to do some kind of semi-supervised learning. In semi-supervised learning, some data points are labelled (you know which class they belong to), and others are not. There ...

Only top scored, non community-wiki answers of a minimum length are eligible