How can I cluster this data frame with several features and observations?

Question

How can I cluster the data frame below with several features and observations? And how would I go about determining the quality of those clusters? Is k-NN appropriate for this?

id     Name             Gender   Dob    Age  Address
1   MUHAMMAD JALIL      Male    1987    33   Chittagong
1   MUHAMMAD JALIL      Male    1987    33   Chittagong
2   MUHAMMAD JALIL      Female  1996    24   Rangpur
2   MRS. JEBA           Female  1996    24   Rangpur
3   MR. A. JALIL        Male    1987    33   Sirajganj
3   MR. A. JALIL        Male    1987    33   Sirajganj
3   MD. A. JALIL        Male    1987    33   Sirajganj
4   MISS. JEBA          Female  1996    24   Rangpur
4   PROF. JEBA          Female  1996    24   Rangpur
1   MD. A. JALIL        Male    1987    33   Chittagong
1   MUHAMMAD A. JALIL   Male    1987    33   Chittagong

Answer 1

A typical clustering algorithm is k-means (and not k-NN, i.e. k-nearest neighbours, which is primarily used for classification). There are other clustering algorithms, such as hierarchical clustering algorithms. sklearn provides functions that implement k-means (and an example), hierarchical clustering algorithms, and other clustering algorithms.

To assess the quality of the produced clusters, you could use the silhouette method (sklearn provides a function that can be used to compute the silhouette score).

Regarding your specific data frame, note that it contains repetitions, so you may want to remove them before starting the clustering procedure. Also, the IDs are not unique, but you probably don't need the IDs for clustering.

Answer 2

Yes you can use KNN algorithm to cluster (well actually its a classification not a clustering if you use KNN) the data. But, first you need to set one feature as a label because KNN is a supervised learning method, it need a labeled data to train the data first. For example you can use Gender as label to classify the data. To determine the quality of the classification result, you can simply use accuracy.

If you don’t want to use a label, you can use unsupervised learning method like K-Means to do the clusters. Because its unsupervised it doesn’t need label so you can use all of the feature to do the clusters task. For the k-means algorithm you can use a library from scikit-learn or create it from scratch. To evaluate the results you can use silhouette score or elbow method (to find the optimal number of cluster).

And don’t forget to do data exploration because maybe it can increase the quality of the cluster results.

You can learn more about the differentiation between K-Means and KNN in the link below: https://pythonprogramminglanguage.com/how-is-the-k-nearest-neighbor-algorithm-different-from-k-means-clustering/

I hope this helps :)

Answer 3

KNN can be used in clustering with the data frame. but there are a number of steps that you must take. 1. You must separate the features you want to cluster. for example you can do clustering dob and age. 2. if there is data of type string you have to change it to an integer. For easier clustering, you can use the Sklearn library. you can access at the following link https://scikit-learn.org/stable/modules/clustering.html

Answer 4

There are several algorithm for clustering such as: K-means, Mean shift, hierarchical,etc. Based on my experience, actually it's K-means(KNN for classifcication).It is suitable for clustering your dataset, there are several steps for clustering your dataset:

1. You have to determine which features that you want cluster
2. Changing your categorical dataset to numerical
3. This step is optional, You can drop columns that are not related to the features you have chosen before
4. Try to to code your clustering (like determine centroid from your dataset, calculate the euclidean distance from your centroid,etc) or if you want to use library maybe sklearn is the right place.

And for determine the quality of your clustering, you can measures SSE(sum of the square error from the items of each cluster),Inter cluster distance,Intra cluster distance for each cluster,Maximum Radius,Average Radius.

Answer 5

you can clustering the data frame with unsupervised algorithm, for example you can use K-Means method. There are some options you can choose to eliminate some features in your data frame, like del dataFrame['Column Name']. In unsupervised learning, the algorithm not calculate the quality of the clusters, but you can set it up by yourself to make a parameter for calculate the quality for each clusters, for example it depend on sum of data in each clusters. Actually you can use KNN algorithm with your data frame, but you need to add a label in there because KNN is a supervised learning, and its function to make a classification, not clustering. hope it useful.

Answer 6

You can use k-nn clustering but you must convert your dataset to numeric values or you can remove the unrelated features in your dataset.