Learn Clustering Method 101 in 5 minutes

In real life we might come across the problem when we want to group our data and learn the basic structure. For example, finding the subgroup of our users can help us come up with more specific marketing strategies.

Another example is that we have a lot of data and want to condense it down to a fewer number of features to be used. The method that solve this kind of problem is clustering.

Clustering is one of the most popular unsupervised approaches. It can identify similar groups of data in a data set. Since there is no objectively “correct” clustering algorithm, the most appropriate clustering algorithm for a particular problem often needs to be chosen experimentally, unless there is a mathematical reason to prefer one cluster model over another.

Connectivity-based clustering, also known as hierarchical clustering, is based on the core idea of objects being more related to nearby objects than to objects farther away.

These algorithms connect “objects” to form “clusters” based on their distance. A cluster can be described largely by the maximum distance needed to connect parts of the cluster.

For example single-linkage clustering use the minimum of object distances(a single distance). The difference between single linkage clustering and K-means is shown above.

Complete linkage clustering however use the maximum of object distances. Good side is clusters will be more condense. Downside is that it still only consider one point. To solve that, we have average linkage clustering.

These methods are not very robust towards outliers, which will either show up as additional clusters or even cause other clusters to merge (known as “chaining phenomenon”, in particular with single-linkage clustering). And methods complexity is high, which makes them too slow for large data sets.

In centroid-based clustering, clusters are represented by a central vector, which may not necessarily be a member of the data set. When the number of clusters is fixed to k, k-means clustering gives a formal definition as an optimization problem: find the k cluster centers and assign the objects to the nearest cluster center, such that the squared distances from the cluster are minimized.

The clustering model most closely related to statistics is based on distribution models. Clusters can then easily be defined as objects belonging most likely to the same distribution. A convenient property of this approach is that this closely resembles the way artificial data sets are generated: by sampling random objects from a distribution.

In density-based clustering, clusters are defined as areas of higher density than the remainder of the data set. Objects in these sparse areas — that are required to separate clusters — are usually considered to be noise and border points.

The most popular density based clustering method is DBSCAN. Similar to linkage based clustering, it is based on connecting points within certain distance thresholds. However, it only connects points that satisfy a density criterion, in the original variant defined as a minimum number of other objects within this radius.

With the recent need to process larger and larger data sets, the willingness to trade semantic meaning of the generated clusters for performance has been increasing.

This led to the development of pre-clustering methods such as canopy clustering, which can process huge data sets efficiently, but the resulting “clusters” are merely a rough pre-partitioning of the data set to then analyze the partitions with existing slower methods such as k-means clustering.

K-means is simple and can be used for a wide variety of data types. But it’s not suitable for all types of data though. It cannot handle non-globular clusters or clusters of different sizes and densities. K-means also has trouble clustering data that contains outliers.

Hierarchical clustering are typically used because the underlying application requires a hierarchy. There also have been some studies that suggest these algorithms can produce better-quality clusters.

However these algorithms are expensive in terms of computational and storage requirements.

DBSCAN is relatively resistant to noise and can handle clusters of arbitrary shapes and sizes. DBSCAN can find many clusters that could not be found using K-means.

However DBSCAN has trouble when the clusters have widely varying densities. It also has trouble with high-dimensional data because density is more difficult to define for such data. Finally ,DBSCAN can be expensive when the computation of nearest neighbors requires computing all pairwise proximities, as it usually the case for high-dimensional data.

Cluster analysis divides data into groups that are meaningful, useful, or both. So in general the usage of cluster analysis can be divide into ‘understanding purpose’ or ‘utility purpose’.

In the context of understanding data, clusters are potential classes and cluster analysis is the study of techniques for automatically finding classes. Below are some examples.

Cluster analysis provides an abstraction from individual data objects to the clusters in which those objects reside. Additionally, some clustering techniques characterize each cluster in terms of a cluster prototype.

These cluster prototypes can be used as the basis for a number of data analysis or data processing techniques. Therefore, in the context of utility, cluster analysis is the study of techniques for finding the most representative cluster prototypes.

Related posts:

The funeral was important and Mr Briggs, the undertaker, made sure everything went well for the burial of the prominent local woman. She had been a businesswoman and, in later years, the Mayoress. Mr…

It is estimated that the human brain makes about 35,000 choices per day on everything from food choices to whether or not you decided to read this blog post! Add on top of that, decisions on your…

Inversion is a simple, yet powerful mental model. Here, I explain how it works and offer some advice on how to use inversion successfully.