5
votes
Accepted
How to refine K-means clustering on a data set?
The usual parameters to adjust in a k-means:
Number of clusters (recall many clusters can have same label).
Distance definition (euclidean is the most basic, Gauss is an
improvement)
Selection of ...
- 1,282
5
votes
What is graph clustering?
In graph clustering, we want to cluster the nodes of a given graph, such that nodes in the same cluster are highly connected (by edges) and nodes in different clusters are poorly or not connected at ...
- 36.4k
3
votes
Accepted
How to compute the number of centroids for K-means clustering algorithm given minimal distance?
Yes, the silhouette method (which is implemented in sklearn as silhouette_score) is commonly used to assess the quality of ...
- 36.4k
3
votes
Accepted
Is unsupervised learning a branch of AI?
There is a problem with confining Artificial Intelligence to a single definition, because it has become an umbrella term encompassing many fields of science. It has come a long way from the "thinking ...
- 3,133
3
votes
How to define machine learning to cover clustering, classification, and regression?
I report three definitions of machine learning (ML) and I also explain that ML can be divided into multiple sub-tasks or sub-categories in this answer. However, it may not always be clear why ...
- 36.4k
2
votes
How can I cluster this data frame with several features and observations?
A typical clustering algorithm is k-means (and not k-NN, i.e. k-nearest neighbours, which is primarily used for classification). There are other clustering algorithms, such as hierarchical clustering ...
- 36.4k
2
votes
How to detect patterns in salary distribution if we are suspecting malicious distribution based on employee's region?
A simple initial approach would be to separate it by position and check for each:
Use linear regression: $\hat{salary} = \sum_i \alpha_i * \hat{region}_i + \sum_k \beta_k * \mathbf{1}[\hat{gender}=k]$...
- 2,329
2
votes
Is this dataset with only two features suitable for clustering with k-means?
One problem with clustering algorithms is that they will typically find you a solution, ie they will split your data set into clusters, but it will find you a structure even if there isn't one. Your ...
- 5,232
2
votes
Could clustering be used to parse pdf documents to get headings and titles?
Yes, you could use clustering: Encode your features as a feature vector and feed it into a clustering algorithm (see Finding Groups in Data for a comprehensive description of these). You could use ...
- 5,232
2
votes
Which metric should I use to assess the quality of the clusters?
You can compute Silhouette Coefficient for your aim. Its values mean:
1: Means clusters are well apart from each other and clearly distinguished.
0: Means clusters are indifferent, or we can say that ...
- 1,703
2
votes
Accepted
What happens if all the features are correlated with each other before clustering?
Essentially, yes. One feature predicts to a reasonably high degree what the other features look like, so the additional features have limited discriminatory power. Obviously there is some effect, as ...
- 5,232
2
votes
Accepted
How to tackle the human error made in labeling datasets for classification tasks like facial expression recognition?
In general the only way to deal with this is by quantifying these labeling mistakes in the output of the model, since the model will learn for them. And in many cases these are not really mistakes, ...
- 1,087
2
votes
Accepted
Interpretation of the Dynamic Time Warping (DTW) graph
The graph plots two things:
the optimal warping path;
the accumulated cost matrix (which looks like a heat map).
To interpret the graph, suppose we draw some other path from bottom left to top right....
- 61
1
vote
Pre-Trained Model for Occupational Coding
My initial (naive) approach would be to
embed the textual data to some representation, using a pre-trained text embedding network. While BERT indeed works, I'd even start off with a simpler embedding,...
- 351
1
vote
Deep Clustering Approach for Unsupervised Video Anomaly Detection
What can be a similar criteria that we can use for pseudo labelling if we use a clustering method instead of an Autoencoder?
Maybe the amount of Data points inside a Cluster. Like you said, an ...
- 51
1
vote
Accepted
Clustering by using Locality sensitive hashing *after* Random projection
I think the following is the way to look at your question.
RP reduces dimensionality based on distance.
LSH clusters data based on a similar distance method used in RP.
The primary function of any ...
- 431
1
vote
Feature Engineering on transactional dataset clustering
The average transaction is a central measure, while the minimum and maximum transactions together give an idea of dispersion. However, these can be very sensitive to individual purchases that might ...
1
vote
How do you evaluate a k-medoids cluster model?
In Finding Groups in Data, Kaufman and Rousseeuw describe ways to evaluate the quality of clusterings. If I remember correctly (it's been some time that I worked with this), for k-means algorithms you ...
- 5,232
1
vote
Accepted
What clustering algorithms work best for datasets with only binary categorical features?
Any clustering algorithm should work -- the main issue is the similarity or distance metric that determines how similar (or different) two elements are. This is often something like Euclidean distance,...
- 5,232
1
vote
How to handle list features in clustering?
You essentially want to have a numerical value to represent the similarity of the lists of two distinct objects. There are a number of metrics to deal with that, eg the Jaccard Index or Dice's ...
- 5,232
1
vote
Is there a clustering algorithm that can make n clusters and the n+1 "others" cluster?
So, I've prepared some data that resembles your sketch:
...
- 1,982
1
vote
Which metric should I use to assess the quality of the clusters?
One more popular metric for this is the Davies Bouldin Score.
You can also take a look at the clustering metrics in scikit documentation.
- 838
1
vote
Accepted
What exactly is the eigenspace of a graph (in spectral clustering)?
In spectral clustering we not find the eigenvectors of a graph (a graph is not a matrix) but the eigenvalues/eigenvectors of the Laplacian matrix related to the adjacency matrix of the graph:
graph =&...
- 1,282
1
vote
How can I cluster this data frame with several features and observations?
Yes you can use KNN algorithm to cluster (well actually its a classification not a clustering if you use KNN) the data. But, first you need to set one feature as a label because KNN is a supervised ...
1
vote
How can I cluster based on the complementary categories?
Note: K-means does not assume an interpretation/label of the clusterings - in fact it is an unsupervised algorithm. The interpretations are a result of human analysis after running K-means.
For ...
- 1,066
1
vote
What is the role of the 'fuzzifier' w in Fuzzy Clustering?
Its not required, you can have $m=1$, actually it can be any number $\geq 1$.
Now the better question is why to have it? The answer is that it adds a smoothing effect. Lets look at it in each of ...
- 2,329
1
vote
How to compute the number of centroids for K-means clustering algorithm given minimal distance?
If you look at Kaufman & Rousseeuw (1990), Finding Groups in Data, they describe an algorithm to evaluate the quality of clusters in agglomerative clustering. You run the clustering algorithm with ...
- 5,232
1
vote
Accepted
How do we know the classification boundaries of the data?
This is the classic question of what structure is or can be. It relates directly to the concepts of generalization, pattern recognition, over-fitting in surface fitting strategies, and learning tabula ...
- 7,345
1
vote
What techniques to explore for dynamic clustering of documents (emails)?
It sounds like you are trying to do some kind of semi-supervised learning. In semi-supervised learning, some data points are labelled (you know which class they belong to), and others are not. There ...
- 9,017
Only top scored, non community-wiki answers of a minimum length are eligible